Mailman

<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.11/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.11/ http://www.mediawiki.org/xml/export-0.11.xsd" version="0.11" xml:lang="en">
  <siteinfo>
    <sitename>Wikitech</sitename>
    <dbname>labswiki</dbname>
    <base>https://wikitech.wikimedia.org/wiki/Main_Page</base>
    <generator>MediaWiki 1.45.0-wmf.4</generator>
    <case>first-letter</case>
    <namespaces>
      <namespace key="-2" case="first-letter">Media</namespace>
      <namespace key="-1" case="first-letter">Special</namespace>
      <namespace key="0" case="first-letter" />
      <namespace key="1" case="first-letter">Talk</namespace>
      <namespace key="2" case="first-letter">User</namespace>
      <namespace key="3" case="first-letter">User talk</namespace>
      <namespace key="4" case="first-letter">Wikitech</namespace>
      <namespace key="5" case="first-letter">Wikitech talk</namespace>
      <namespace key="6" case="first-letter">File</namespace>
      <namespace key="7" case="first-letter">File talk</namespace>
      <namespace key="8" case="first-letter">MediaWiki</namespace>
      <namespace key="9" case="first-letter">MediaWiki talk</namespace>
      <namespace key="10" case="first-letter">Template</namespace>
      <namespace key="11" case="first-letter">Template talk</namespace>
      <namespace key="12" case="first-letter">Help</namespace>
      <namespace key="13" case="first-letter">Help talk</namespace>
      <namespace key="14" case="first-letter">Category</namespace>
      <namespace key="15" case="first-letter">Category talk</namespace>
      <namespace key="110" case="first-letter">Obsolete</namespace>
      <namespace key="111" case="first-letter">Obsolete talk</namespace>
      <namespace key="112" case="first-letter">OfficeIT</namespace>
      <namespace key="113" case="first-letter">OfficeIT talk</namespace>
      <namespace key="116" case="first-letter">Tool</namespace>
      <namespace key="117" case="first-letter">Tool talk</namespace>
      <namespace key="498" case="first-letter">Nova Resource</namespace>
      <namespace key="499" case="first-letter">Nova Resource Talk</namespace>
      <namespace key="666" case="first-letter">Heira</namespace>
      <namespace key="667" case="first-letter">Heira Talk</namespace>
      <namespace key="710" case="first-letter">TimedText</namespace>
      <namespace key="711" case="first-letter">TimedText talk</namespace>
      <namespace key="828" case="first-letter">Module</namespace>
      <namespace key="829" case="first-letter">Module talk</namespace>
    </namespaces>
  </siteinfo>
  <page>
    <title>Mailman</title>
    <ns>0</ns>
    <id>3208</id>
    <revision>
      <id>2309610</id>
      <parentid>2303531</parentid>
      <timestamp>2025-06-07T13:19:24Z</timestamp>
      <contributor>
        <username>Taavi</username>
        <id>13997</id>
      </contributor>
      <comment>/* Upgrading Mailman3 */ update link</comment>
      <origin>2309610</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="11360" sha1="0vwu0qs5l5glneg37lmk2fhoo1jmv1x" xml:space="preserve">{{TOCright}}

== See also ==
This page is mainly for the developers and global-list-admins. You may be looking for:
* User information, including how to request a new list: [[meta:Mailing lists]]
* Documentation for the admins of specific mailing-lists: [[meta:Mailing lists/Administration]]
* Historical mailman2 operations: [[Mailman/Mailman2]]


== Components ==
Mailman3 has multiple components. Each doing a different thing. This is a simplified overview. For a more visual diagram, see the [https://docs.mailman3.org/en/latest/architecture.html upstream architecture overview].
* mailman3 is the service that handles emails and talks to MTA (in our case exim4) through LMTP/SMTP
* mailman3-web or postorius is a django-based web service that handles settings, subscriptions, etc. It doesn't do anything on its own (except user management) but it talks internally to the mailman3 service through REST API.
* hyperkitty (being used in the same webservice) is responsible for archives. It handles search, etc.
* Search backend: Is xapian and the search index is in /var/lib/mailman3/web/fulltext_xapian_index
* Databases: Two databases both in m5:
** mailman3 which mailman's core service talks to. It holds memberships, lists, etc.
** mailman3web which postorius and hyperkitty talk to. It holds archive of all emails and is rather big.
** MTA: It's exim4 (see [[Mailman/MTA]] for more information)

== {{Anchor|Who has access}} Who has access? ==

=== Mailman3 ===
The following people have &quot;superuser&quot; access via the web UI for admin access to all lists:

* [[m:User:Legoktm|Legoktm]]
* [[m:User:Ladsgroup|Ladsgroup]]
* [[User:Quiddity|Quiddity]] (Nick Wilson) (WMF)
* &quot;Administrator&quot; (all SRE members or users with root)
* &quot;WMFOffice&quot; (Trust and Safety team)

== How To ==

=== Create a mailing list ===
* Go to https://lists.wikimedia.org/postorius/lists/new/ (as an administrator)
* or ssh to lists1001.wikimedia.org and run &quot;sudo mailman-wrapper create --owner OWNER LISTNAME@lists.wikimedia.org&quot;

Consider if you need to notify [https://office.wikimedia.org/wiki/Trust_and_Safety Trust and Safety] when creating new mailing lists.

The only part of new mailing lists that can't be easily changed is its name. Pleas make sure it follows [[m:Mailing lists/Standardization|naming standards]] before creating.

=== Disable or re-enable a mailing list ===
* Remove owners and add disabled-lists@lists.wikimedia.org as owner
** This is important, otherwise we can't find it and change/fix its settings.
* Set all emails to be discarded (both member and non-member)
* Ban &lt;code&gt;^.*@.*&lt;/code&gt;
* Add to description that this mailing list is disabled like [Disabled] or [Archived]

=== Remove a mailing list ===
Note: unless you know what you're doing, don't actually do this. It most likely does not do what you want. Just disable the list (see above) instead. 
* Go to https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/delete (as an administrator)
* or ssh to lists1001.wikimedia.org and run &quot;sudo mailman-wrapper remove LISTNAME@lists.wikimedia.org&quot;

=== Remove a message from the mailing list archives ===
;Delete the whole thread
* Go to the thread in hyperkitty ([https://lists.wikimedia.org/hyperkitty/list/abstract-wikipedia@lists.wikimedia.org/thread/VPVALIXWEBAJDJYRI3EJX7HCGWXRFYFD/ example]).
* Click on &quot;Delete this thread&quot;

;Delete a single message
* Go to the message in hyperkitty (this can be tricky, go to the thread, click on the chain icon aka &quot;Permanant link to this email&quot;). [https://lists.wikimedia.org/hyperkitty/list/abstract-wikipedia@lists.wikimedia.org/message/VPVALIXWEBAJDJYRI3EJX7HCGWXRFYFD/ example]
* Click on &quot;Delete this message&quot;
=== Export a listing of all subscribers to a mailing list ===
* Go to members settings: https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/members/member/
* Click on &quot;CSV export&quot;

=== Check if an email address is subscribed to any lists ===

=== Remove an individual from all mailing lists ===
* ssh to lists1001.wikimedia.org
* Run &quot;sudo mailman-wrapper delmembers -m EMAIL_ADDRESS --fromall&quot;

=== Reset a user's password ===
* ssh to lists1001.wikimedia.org.
* Run &quot;sudo mailman-web changepassword USERNAME&quot;

=== Rename a mailing list ===


=== Import from Google Groups ===
It is straightforward to migrate a list from Google Groups to Mailman. A list owner in Google should use Google Takeout ([https://support.google.com/groups/answer/9975859 see documentation]) to export the member list (members.csv) and archives (topics.mbox). File a task requesting a new list and include a link to the Takeout result (if the list contents are private, provide this information to the Mailman admins/SRE team privately).

Create the new list in Mailman like normal, then use the &quot;Mass subscription&quot; form at &lt;https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/mass_subscribe/&gt; to import the existing subscribers.

Copy the mbox to the mailing list server and run:

&lt;code&gt;sudo mailman-web hyperkitty_import -l LISTNAME@lists.wikimedia.org /path/to/topics.mbox&lt;/code&gt;

This might take a while because of the &quot;Synchronizing properties with Mailman&quot; step, that's fine.

Then update the search index:

&lt;code&gt;sudo mailman-web update_index_one_list LISTNAME@lists.wikimedia.org&lt;/code&gt;

All done! Make sure to have the list owner in Google disable the group there, and remind subscribers to update the lists' email address to LISTNAME@'''lists.'''wikimedia.org.

=== Add a list to Gmane ===
A new list can be added to Gmane very easily if the first message has not already been sent or history is not important. If the history needs to be imported, a ticket should be file in Phabricator with projects 'Wikimedia-Mailing-list' and 'operations' ([https://phabricator.wikimedia.org/maniphest/task/create/?projects=Wikimedia-Mailing-lists%20operations easy link] stating which list needs to imported and to which Gmane group if it exists already.


=== Docs and links ===
* [https://www.gnu.org/software/mailman/site.html Site Administrator docs on gnu.org]

=== Authorized recipients for ops@lists.wikimedia.org ===
Anyone with a signed NDA can be on the ops@lists.wikimedia.org mailing list.
For a list of signed NDAs for staff/contractors there is [[phab:T83783|T83783]], [[phab:project/members/61|WMF-NDA]] and &lt;tt&gt;ldap/nda&lt;/tt&gt; (or &lt;tt&gt;ldap/wmf&lt;/tt&gt;).

=== Migrate servers ===
See [[Mailman/Migration]] for comments made about past migrations (lily to sodium, sodium to fermium, fermium to lists1001).

=== Upgrading Mailman3 ===
When upgrading Mailman3 packages for a version that includes schema changes, use the following process.

# Downtime host in icinga
# Disable puppet
# Manually stop &lt;code&gt;mailman3&lt;/code&gt; and &lt;code&gt;mailman3-web&lt;/code&gt; systemd services
# Run apt to install the new packages
# Run &lt;code&gt;sudo mailman-wrapper help&lt;/code&gt; to apply the mailman3 schema updates (yes, running just the &quot;help&quot; command will apply the schema changes)
# Run &lt;code&gt;sudo mailman-wrapper help&lt;/code&gt; again to make sure the schema update didn't obviously break anything.
# Run &lt;code&gt;sudo mailman-web migrate&lt;/code&gt; to apply the mailman3 schema updates
#Clear static CSS/JS caches: &lt;code&gt;sudo mailman-web collectstatic --clear --noinput &amp;&amp; sudo mailman-web compress&lt;/code&gt;
# Start mailman3 and mailman3-web systemd services again, keep an eye on errors in the logs
# Re-enable puppet and remove icinga downtime if it hasn't expired yet.
Post upgrade:

* Verify that the messages the [[git:mediawiki/core/+/refs/heads/master/includes/installer/Task/MailingListSubscribeTask.php|MediaWiki Installer screen-scrapes]] haven't changed.

=== Block abusive traffic ===
* Use &lt;code&gt;profile::lists::web_deny_conditions&lt;/code&gt; in private puppet to block at the Apache level (see [[gerrit:736211]])
* Use the standard &lt;code&gt;abuse_networks&lt;/code&gt; in private puppet which will block at the ferm/network level (see [[gerrit:736552]])

== Configuration details ==
The new Mailman3 setup lives on [[fermium|lists1001]]. Some state is under &lt;tt&gt;/var/lib/mailman3/&lt;/tt&gt;, the global configuration is in &lt;tt&gt;/etc/mailman/&lt;/tt&gt; via Puppet.

The mail server used is [[Exim]], the web server used is [[Apache]].

=== Packages ===
We are currently using [https://apt-browser.toolforge.org/buster-wikimedia/component/mailman3/ custom backported packages] of the mailman3 suite. A few patches have been cherry-picked from upstream to fix issues we've run into. The packaging Git repositories can be found on apt1001 in legoktm's home directory.

There is currently [[phab:T282348#7125322|one change]] that is hotpatched and not in the package yet.

=== Mailman setup ===
TODO: Add

=== Mail server setup ===
See [[Mailman/MTA]]

=== Web server setup ===
To get Mailman running with Apache the puppet class role::lists is applied which uses module mailman.
Puppet module mailman uses puppet module Apache which sets up Apache and the site config is in the template lists.wikimedia.org.erb.

=== Backups ===
[[fermium|lists1001]] is backed up to [[backup1001]] using [[Bacula]]. All of /var/lib/mailman3 is backed up, which potentially includes on-disk templates and the search index. We do not back up /var/lib/mailman3/web which comes from the package and digests that haven't been sent yet.

The database lives on m5, which is also regularly backed up into Bacula, too.

=== Tested failure modes ===
Because mail delivery and transport should be reliable, I have tested what happens in certain failure modes, e.g. when SpamAssassin's spamd daemon is not running.

==== Spamd not running ====
Because of the &lt;tt&gt;/defer_ok&lt;/tt&gt; modifiers in the Exim ACLs, Exim will act as if no spam filtering attempts are made when spamd is not running, and will accept the message. The following lines are logged:
 spam acl condition: warning - spamd connection to 127.0.0.1, port 783 failed: Connection refused
 spam acl condition: all spamd servers failed
 H=xxx.xxxxxxx.xx [xx.xx.xx.xx]:xxxx I=[145.97.39.157]:25 U=exim Warning: ACL &quot;warn&quot; statement skipped: condition test deferred

==== Mailman not running ====
If the Mailman queue runner daemons are not running, incoming messages will still get delivered to the Mailman queue by Exim. However, nothing else will happen until the Mailman processes are started.

== Spam fighting ==
See [[Mailman/Spam fighting]]

== Templates ==
Most emails sent out by Mailman3 can be customized using &quot;templates&quot;. You can see the [[gerrit:plugins/gitiles/operations/software/mailman-templates/+/refs/heads/master/en/|default English templates]]. You can edit them by visiting [https://lists.wikimedia.org/postorius/lists/YOURLISTNAME.lists.wikimedia.org/templates https://lists.wikimedia.org/postorius/lists/&lt;code&gt;YOURLISTNAME&lt;/code&gt;.lists.wikimedia.org/templates]. These are [[betawiki:Special:Translate/wikimedia-mailman-templates|translatable on translatewiki.net]].

It may also be useful to read the [https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/templates.html#line-wrapping template documentation] (the code parts can be ignored).

== Monitoring ==
See [[Mailman/Monitoring]]

== Mailman2 data ==
Most private Mailman2 data is backed up in bacula in a long-term backup ([[phab:T282303#7114284|details]]). See [[Bacula#Restore_(aka_Panic_mode)|how to restore a file]].

[[Category:Mail]]
[[Category:Services]]</text>
      <sha1>0vwu0qs5l5glneg37lmk2fhoo1jmv1x</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Toolsbeta/SAL</title>
    <ns>498</ns>
    <id>6726</id>
    <revision>
      <id>2309611</id>
      <parentid>2309562</parentid>
      <timestamp>2025-06-07T16:49:57Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>dcaro: extend the volume toolforge-prometheus-a to 20G</comment>
      <origin>2309611</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="466299" sha1="hb4nb086xp3h99x7r1pl7zv973suxny" xml:space="preserve">=== 2025-06-07 ===
* 16:49 dcaro: extend the volume toolforge-prometheus-a to 20G

=== 2025-06-06 ===
* 18:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli
* 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-06-05 ===
* 14:43 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:30 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-06-04 ===
* 00:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-06-02 ===
* 23:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 23:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:05 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 18:01 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-22 ===
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 19:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-6
* 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-6
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-5
* 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-prometheus-1
* 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1

=== 2025-05-21 ===
* 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud
* 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-20 ===
* 18:24 bd808: Made addshore an admin

=== 2025-05-19 ===
* 08:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2025-05-16 ===
* 18:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
* 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase

=== 2025-05-15 ===
* 08:13 taavi: renew expiring Puppet CA cert

=== 2025-05-14 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2025-05-13 ===
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-12 ===
* 19:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 18:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 15:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:44 taavi: fix security groups for frontproxy-nginx metricsinfra job
* 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 08:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-05-09 ===
* 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 22:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
* 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-05-08 ===
* 17:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 17:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 16:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 15:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 13:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:53 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2025-05-07 ===
* 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 09:36 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]]
* 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-05-06 ===
* 12:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-24 ===
* 18:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 18:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli

=== 2025-04-23 ===
* 15:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 15:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-21 ===
* 10:13 taavi: update cluster-info config map to use k8s.svc.toolsbeta.eqiad1.wikimedia.cloud service name [[phab:T262562|T262562]]

=== 2025-04-17 ===
* 16:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-04-16 ===
* 19:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 19:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission

=== 2025-04-15 ===
* 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:28 arturo: added `toolsbeta-tofu` bot account with `member` permissions [[phab:T391474|T391474]]

=== 2025-04-11 ===
* 21:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 19:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-09 ===
* 10:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-04-08 ===
* 01:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-04-07 ===
* 20:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 19:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 19:00 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 18:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 08:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 07:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 07:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 07:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 06:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 04:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-04-04 ===
* 09:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 09:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 09:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 08:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 08:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 07:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 07:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 07:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 07:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 07:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 07:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 06:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-31 ===
* 14:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 14:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:30 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:24 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:13 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:11 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 12:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])
* 11:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]])

=== 2025-03-25 ===
* 15:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 08:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2025-03-24 ===
* 18:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-20 ===
* 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
* 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'

=== 2025-03-13 ===
* 22:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 22:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 16:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli
* 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-12 ===
* 19:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]])
* 15:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-builder
* 15:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 03:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
* 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-11 ===
* 18:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]])
* 17:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]])
* 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]])
* 17:35 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]])
* 17:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]])
* 17:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]])
* 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]])
* 17:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]])
* 14:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]])
* 14:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli
* 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 14:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
* 10:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission

=== 2025-03-10 ===
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 18:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 18:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 18:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 17:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 17:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-03-06 ===
* 10:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-05 ===
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-03-04 ===
* 21:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 21:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 20:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 14:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 11:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 11:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 09:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission

=== 2025-03-03 ===
* 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 12:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 11:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 09:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2025-02-27 ===
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2025-02-26 ===
* 19:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 19:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli
* 10:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-02-24 ===
* 20:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 20:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 20:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-02-19 ===
* 17:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]])
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]])

=== 2025-02-17 ===
* 17:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2025-02-06 ===
* 17:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 14:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer
* 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer
* 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 12:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 12:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2025-02-01 ===
* 15:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 15:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:14 andrewbogott: hard rebooting all VMs for [[phab:T385264|T385264]]
* 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes

=== 2025-01-29 ===
* 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 00:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2025-01-23 ===
* 21:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 20:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T370245|T370245]])
* 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T370245|T370245]])
* 14:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 14:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-22 ===
* 18:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 18:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-21 ===
* 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 16:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 16:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 16:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14
* 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14
* 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 15:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]])
* 12:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9
* 12:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-8
* 12:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7
* 12:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5
* 12:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5
* 12:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10
* 12:40 andrewbogott: rebooting toolsbeta-nfs-3 and then restarting all k8s-nfs workers
* 12:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10

=== 2025-01-20 ===
* 13:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli
* 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli

=== 2025-01-17 ===
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-15 ===
* 04:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:36 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 03:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2025-01-07 ===
* 00:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico
* 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 00:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 00:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics
* 00:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics
* 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 00:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2025-01-06 ===
* 23:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 23:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor
* 23:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 23:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 23:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 23:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 23:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 23:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 16:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor

=== 2024-12-13 ===
* 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 13:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-12-06 ===
* 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 07:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 07:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-05 ===
* 16:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 13:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-12-04 ===
* 19:37 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 19:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 19:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 17:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 17:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 17:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 16:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 16:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 16:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 15:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 14:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 14:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 13:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 13:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:38 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 13:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 13:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 01:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-12-03 ===
* 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 21:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:10 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 21:02 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
* 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 20:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 19:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:04 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 18:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 17:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-29 ===
* 08:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 08:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 08:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 07:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 07:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 06:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 05:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
* 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
* 03:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 00:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 00:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-11-25 ===
* 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:40 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 10:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-23 ===
* 07:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]])
* 07:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]])

=== 2024-11-20 ===
* 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 14:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 12:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 11:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 00:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]])
* 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]])

=== 2024-11-19 ===
* 21:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 21:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 20:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
* 20:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 20:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362867|T362867]])
* 20:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]])
* 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]])
* 20:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]])
* 20:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]])
* 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]])
* 19:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]])
* 19:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 19:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission ([[phab:T362867|T362867]])
* 19:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]])
* 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:10 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice
* 10:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice

=== 2024-11-18 ===
* 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 10:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer

=== 2024-11-14 ===
* 16:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
* 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 16:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-12 ===
* 13:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:41 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 13:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 09:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-11-11 ===
* 17:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 17:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 16:04 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 16:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:27 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 15:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics
* 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-11-07 ===
* 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-11-06 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 16:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 07:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 07:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 07:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 07:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-11-05 ===
* 17:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
* 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 08:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 07:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 07:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

=== 2024-11-04 ===
* 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 16:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 12:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 11:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 11:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 11:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 11:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 10:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-30 ===
* 15:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:33 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:20 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 14:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])
* 13:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]])

=== 2024-10-29 ===
* 09:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta in eqiad1
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.create_project for project toolsbeta in eqiad1

=== 2024-10-16 ===
* 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-15 ===
* 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 16:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-10-10 ===
* 08:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-10-09 ===
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-10-08 ===
* 17:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 16:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 16:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld ([[phab:T376710|T376710]])
* 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]])
* 08:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
* 08:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 08:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain_kubeusers
* 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain_kubeusers

=== 2024-10-04 ===
* 11:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 11:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 11:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 11:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-10-03 ===
* 14:04 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) [[phab:T374908|T374908]]
* 14:03 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531)

=== 2024-10-01 ===
* 10:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 10:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

=== 2024-09-28 ===
* 00:06 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 00:01 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

=== 2024-09-27 ===
* 23:51 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 23:44 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-26 ===
* 16:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 16:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 15:57 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 15:51 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T359641|T359641]])
* 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T359641|T359641]])
* 10:20 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
* 10:12 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
* 10:04 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
* 09:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
* 07:59 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli
* 07:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
* 07:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
* 07:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
* 06:44 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
* 06:43 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

=== 2024-09-25 ===
* 14:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-10
* 08:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:24 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 07:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 07:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7
* 07:02 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:55 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:48 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:23 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 06:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 06:06 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:50 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:49 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 05:48 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10
* 05:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10
* 05:33 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:32 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the toolsbeta cluster
* 05:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 05:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 05:16 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 05:15 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 04:42 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 04:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-24 ===
* 22:03 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]])
* 21:41 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]])
* 21:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]])

=== 2024-09-21 ===
* 03:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7
* 03:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7
* 03:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7

=== 2024-09-20 ===
* 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]])
* 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 19:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]])
* 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]])
* 00:30 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]])
* 00:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]])

=== 2024-09-19 ===
* 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]])
* 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]])
* 17:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]])
* 14:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]])
* 14:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])
* 14:10 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]])
* 14:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]])

=== 2024-09-11 ===
* 12:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 12:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 12:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 12:26 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 12:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 11:44 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:34 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 09:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0)
* 09:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:24 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 09:09 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 08:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-09-10 ===
* 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:46 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:35 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-6.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])
* 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 14:21 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]])

=== 2024-09-09 ===
* 16:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
* 16:09 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 14:29 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-06 ===
* 09:17 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:14 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:13 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 09:10 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 09:00 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 08:55 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 08:34 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 08:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 08:28 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 08:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 06:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

=== 2024-09-05 ===
* 20:51 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager
* 20:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager
* 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-9
* 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 17:39 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 17:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 17:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-8
* 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8
* 17:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-7
* 17:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-7
* 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])
* 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]])

=== 2024-09-04 ===
* 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 13:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 13:55 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 13:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 11:20 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

=== 2024-09-03 ===
* 20:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 19:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 19:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
* 19:40 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
* 19:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 19:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 19:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
* 19:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 19:07 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 19:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 18:50 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 16:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:53 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 16:40 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 16:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 16:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
* 15:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:58 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component kyverno
* 14:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
* 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
* 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
* 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-09-02 ===
* 09:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 09:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 09:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
* 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
* 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-29 ===
* 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 16:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder

=== 2024-08-28 ===
* 17:22 andrewbogott: shutting down toolsbeta-harbor-2 to (I hope) quiet alerts. Raymond can start this up again when he's back.
* 14:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15
* 14:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15
* 14:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
* 13:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
* 13:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
* 13:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15
* 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15
* 13:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15
* 06:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx
* 06:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx
* 06:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
* 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

=== 2024-08-27 ===
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
* 08:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico

=== 2024-08-26 ===
* 09:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-21 ===
* 05:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 05:31 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 05:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
* 05:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 05:13 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder
* 05:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
* 04:52 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 04:45 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
* 04:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
* 04:03 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
* 03:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
* 03:41 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 03:35 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 03:12 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 03:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 02:53 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 02:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:54 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 01:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 01:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 01:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.run_tests
* 01:39 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 01:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

=== 2024-08-13 ===
* 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

=== 2024-08-12 ===
* 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 12:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
* 11:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 11:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
* 10:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
* 10:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
* 10:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 10:01 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
* 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
* 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:14 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
* 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
* 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
* 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

=== 2024-08-08 ===
* 16:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
* 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
* 16:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
* 16:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
* 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components
* 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components
* 15:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component compontents
* 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component compontents

=== 2024-08-06 ===
* 13:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 13:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 13:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 12:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

=== 2024-08-05 ===
* 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 18:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 18:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 18:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api
* 17:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 17:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
* 17:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
* 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:14 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=1)
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 15:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
* 14:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
* 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests
* 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
* 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-08-01 ===
* 15:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-31 ===
* 20:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 20:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:52 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 12:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 11:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
* 11:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

=== 2024-07-30 ===
* 17:34 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli

=== 2024-07-29 ===
* 18:22 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 18:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
* 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
* 12:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
* 12:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli
* 12:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli
* 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 11:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
* 09:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

=== 2024-07-25 ===
* 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 08:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-07-24 ===
* 08:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 06:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 06:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2024-07-23 ===
* 14:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 12:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 12:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-22 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-18 ===
* 14:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 08:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-17 ===
* 14:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-api
* 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 08:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-07-16 ===
* 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 08:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-15 ===
* 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 07:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-12 ===
* 10:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16
* 10:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16
* 10:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
* 09:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
* 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
* 09:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16
* 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16

=== 2024-07-11 ===
* 17:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16
* 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16
* 12:10 arturo: upgrading k8s cluster to 1.25 (control plane) [[phab:T369168|T369168]]
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.24.17 to 1.25.16
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

=== 2024-07-10 ===
* 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 15:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:48 arturo: manually deleted tool-test8 and tool-test8xx k8s namespaces to have them recreated by maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-07-09 ===
* 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 14:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 14:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-07-08 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 13:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 13:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 12:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 11:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-07-05 ===
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 01:42 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 01:41 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
* 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-07-04 ===
* 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 17:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component api-gateway
* 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]]
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-07-03 ===
* 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 09:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2024-07-02 ===
* 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
* 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:54 arturo: cleanup extra redundant cert-signing settings from controller-manager arguments
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 10:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-07-01 ===
* 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-06-28 ===
* 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 09:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-06-27 ===
* 16:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-26
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-26
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-25
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-25
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
* 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
* 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=97) for server toolsbeta-test-k8s-etcd-23
* 14:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 14:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 14:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23
* 10:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:30 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]])
* 09:28 arturo: disabled PodSecurityPolicy admission plugin from apiserver static pod manifests ([[phab:T368142|T368142]])
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-26 ===
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:17 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]])
* 09:15 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]])
* 09:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-25 ===
* 12:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-5'
* 11:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migirate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migirate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-6'
* 09:42 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]])
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-24 ===
* 15:44 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]])
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-21 ===
* 03:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd

=== 2024-06-20 ===
* 14:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 09:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-19 ===
* 09:55 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
* 04:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 04:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 04:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 04:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-17 ===
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-ingress-7
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-ingress-7
* 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-worker-10
* 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-worker-10
* 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-haproxy-5
* 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-haproxy-5
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-legacy-redirector-2
* 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-harbor-1
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-harbor-1
* 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetserver-1
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetserver-1
* 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetdb-03
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetdb-03
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-6
* 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-5
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-5
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-prometheus-1
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-mail-2
* 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-mail-2
* 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-bastion-6
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-bastion-6
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-docker-imagebuilder-2
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-docker-imagebuilder-2
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-acme-chief-2
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-static-2
* 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-static-2

=== 2024-06-14 ===
* 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-sgebastion-05
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-sgebastion-05
* 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-redis-1
* 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-redis-1
* 08:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-12 ===
* 17:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 17:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 17:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 16:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 15:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-11 ===
* 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-07 ===
* 11:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
* 11:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
* 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-06 ===
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-06-05 ===
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-06-04 ===
* 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-06-03 ===
* 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 09:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 08:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 08:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

=== 2024-05-30 ===
* 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-29 ===
* 14:56 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 03:00 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 03:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-28 ===
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-27 ===
* 16:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-25 ===
* 21:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 21:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 20:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 20:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-23 ===
* 13:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-05-15 ===
* 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-05-14 ===
* 13:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

=== 2024-05-10 ===
* 13:57 taavi: renew k8s prometheus certificate

=== 2024-05-07 ===
* 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-05-06 ===
* 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 07:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2024-05-04 ===
* 15:16 taavi: $ sudo docker exec -it striker-toolsbeta.service poetry run python3 manage.py loaddata software_license.json
* 14:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2024-05-03 ===
* 15:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-30 ===
* 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-26 ===
* 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-04-25 ===
* 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-24 ===
* 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-18 ===
* 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-15 ===
* 20:26 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 20:26 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 18:21 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 18:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:51 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:50 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 17:31 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 17:30 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
* 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 11:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-12 ===
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 15:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
* 10:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
* 09:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 09:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component volume-admisison
* 09:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admisison
* 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 05:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 04:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 04:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 03:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 02:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
* 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
* 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
* 01:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
* 01:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
* 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
* 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
* 01:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 01:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 01:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 01:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node

=== 2024-04-11 ===
* 23:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 08:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-04-10 ===
* 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node
* 02:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 02:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 02:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 02:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 00:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-04-09 ===
* 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 23:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 23:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 22:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 22:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 20:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 18:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 13:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-08 ===
* 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-05 ===
* 12:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-04-03 ===
* 16:05 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 14:30 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-04-02 ===
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 18:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 18:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 17:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-localdisk
* 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-localdisk
* 15:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-registry-02
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-registry-02

=== 2024-04-01 ===
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 15:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 14:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node

=== 2024-03-28 ===
* 17:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
* 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 16:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
* 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T349207|T349207]])
* 14:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]])
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]])
* 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]])
* 14:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-3
* 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-3
* 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-4
* 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-proxy'
* 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'toolsbeta-proxy'
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'

=== 2024-03-27 ===
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-2
* 12:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-2

=== 2024-03-26 ===
* 14:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0)
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0)
* 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 14:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:56 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:55 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97)
* 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server
* 13:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3
* 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3
* 13:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:31 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
* 13:31 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service
* 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 13:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99)
* 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server

=== 2024-03-25 ===
* 18:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-legacy-redirector
* 18:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-legacy-redirector

=== 2024-03-22 ===
* 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-21 ===
* 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-4
* 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-4
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-3
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-3
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

=== 2024-03-20 ===
* 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
* 11:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-19 ===
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-03-18 ===
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-static-1
* 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-static-1
* 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
* 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-03-16 ===
* 11:09 taavi: reenable puppet on toolsbeta-test-k8s-control-7/8

=== 2024-03-15 ===
* 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-14 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-imagebuilder-01
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-imagebuilder-01
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]])
* 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]])
* 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 09:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])

=== 2024-03-13 ===
* 16:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 16:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]])
* 15:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 15:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-12 ===
* 11:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])
* 11:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]])

=== 2024-03-11 ===
* 16:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 16:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2024-03-07 ===
* 14:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 13:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-05 ===
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 16:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-03-04 ===
* 17:55 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 17:55 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-03-01 ===
* 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 21:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2024-02-28 ===
* 00:39 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 00:39 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-26 ===
* 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-02-22 ===
* 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 13:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-02-21 ===
* 17:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 14:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
* 13:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-20 ===
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-control-6
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
* 13:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 13:46 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:38 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-test-k8s-worker-9
* 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 13:26 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1
* 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
* 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1

=== 2024-02-19 ===
* 18:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-02-15 ===
* 11:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
* 11:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
* 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
* 10:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-02-13 ===
* 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-4
* 14:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-5
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-4
* 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-4
* 10:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 10:11 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-3
* 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-3
* 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
* 09:59 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 09:50 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-8
* 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-7
* 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-7

=== 2024-02-12 ===
* 10:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2024-02-09 ===
* 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2024-02-08 ===
* 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
* 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:30 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-6
* 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeat-test-k8s-worker-6
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeat-test-k8s-worker-6
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-10
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-10

=== 2024-02-06 ===
* 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]])
* 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]])

=== 2024-02-05 ===
* 09:55 arturo: grant myself member and admin privileges

=== 2024-01-31 ===
* 13:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-29 ===
* 13:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud
* 13:04 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud

=== 2024-01-26 ===
* 10:59 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
* 10:59 wmbot~taavi@runko: Added a new k8s control toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
* 10:47 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
* 10:43 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
* 10:42 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster

=== 2024-01-25 ===
* 12:30 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:30 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:27 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
* 12:24 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
* 11:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-24 ===
* 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster

=== 2024-01-23 ===
* 19:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 19:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-19 ===
* 15:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 12:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-17 ===
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-12 ===
* 09:22 taavi: upgrade prometheus on toolsbeta-prometheus-1

=== 2024-01-11 ===
* 17:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-09 ===
* 17:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 17:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2024-01-08 ===
* 10:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 10:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2024-01-05 ===
* 14:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:50 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 11:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 11:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-12-26 ===
* 19:15 dhinus: hard reboot toolsbeta-bastion-6 as it's unreachable

=== 2023-12-20 ===
* 18:51 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase
* 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
* 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase

=== 2023-12-15 ===
* 13:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T341067|T341067]])
* 13:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T341067|T341067]])

=== 2023-12-13 ===
* 16:23 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=97)
* 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec
* 14:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder ([[phab:T352774|T352774]])
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder ([[phab:T352774|T352774]])
* 13:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T338142|T338142]])
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T338142|T338142]])
* 10:44 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T338142|T338142]])
* 10:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T338142|T338142]])
* 09:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 09:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-12-12 ===
* 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])

=== 2023-12-11 ===
* 19:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 19:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 15:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 15:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 15:23 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api ([[phab:T352774|T352774]])
* 15:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]])
* 13:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]])
* 13:32 dcaro: rebooted the bastion-6, did not seem to have network and was failing to mount nfs
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 13:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 13:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T352774|T352774]])
* 13:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T352774|T352774]])

=== 2023-12-07 ===
* 14:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-05 ===
* 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 21:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 17:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 17:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
* 09:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-12-04 ===
* 09:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 09:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-12-01 ===
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-11-23 ===
* 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-22 ===
* 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 09:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 09:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-11-20 ===
* 15:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-17 ===
* 15:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes
* 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes
* 14:57 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:57 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 14:56 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

=== 2023-11-09 ===
* 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 10:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-11-01 ===
* 09:06 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=99)
* 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-30 ===
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-27 ===
* 09:41 dcaro: resizing toolsbeta-prometheus-1 to 4 cores, 8Gram
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
* 09:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-10-26 ===
* 09:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
* 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

=== 2023-10-25 ===
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
* 10:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6
* 10:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6
* 10:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
* 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster

=== 2023-10-23 ===
* 15:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 15:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-20 ===
* 10:37 blancadesal: harbor up again and upgraded from 2.5 to 2.9 ([[phab:T346241|T346241]])
* 10:11 dcaro: taking harbor down for upgrade ([[phab:T346241|T346241]])

=== 2023-10-18 ===
* 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

=== 2023-10-13 ===
* 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 09:06 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=97)
* 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-12 ===
* 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
* 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

=== 2023-10-10 ===
* 08:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 08:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-09 ===
* 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-05 ===
* 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

=== 2023-10-04 ===
* 16:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 16:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 16:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 16:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 14:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
* 07:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 07:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

=== 2023-10-03 ===
* 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
* 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
* 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
* 09:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

=== 2023-09-27 ===
* 14:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
* 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
* 14:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
* 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
* 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

=== 2023-09-25 ===
* 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-20 ===
* 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

=== 2023-09-19 ===
* 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
* 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-15 ===
* 12:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
* 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

=== 2023-09-14 ===
* 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
* 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
* 12:05 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer
* 12:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer
* 11:59 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission
* 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission
* 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 11:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 10:16 dcaro: deploy bulids-api 0.0.96
* 09:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 09:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
* 08:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
* 08:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

=== 2023-09-13 ===
* 16:41 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 16:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:27 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]])
* 10:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone
* 10:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone

=== 2023-09-11 ===
* 16:05 dcaro: deploy builds-builder ([[phab:T341084|T341084]])
* 11:36 dcaro: deploy kubernetes-metrics ([[phab:T341084|T341084]])

=== 2023-09-06 ===
* 08:47 arturo: switch project to new DNS recursor via horizon project hiera ([[phab:T345240|T345240]], [[phab:T342621|T342621]])

=== 2023-09-05 ===
* 13:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]])
* 13:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]])

=== 2023-08-31 ===
* 15:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0)
* 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 15:38 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 12:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 12:42 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_job_logs
* 12:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
* 09:36 wm-bot2: deployed kubernetes component api-gateway ({{Gerrit|c0faf0f}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
* 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
* 08:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
* 08:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
* 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
* 08:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17

=== 2023-08-30 ===
* 11:18 wm-bot2: toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:17 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 11:15 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo
* 10:05 dcaro: upgrade toolforge-weld to 1.2.1 ([[phab:T344155|T344155]])
* 08:15 taavi: updating toolsbeta k8s cluster to 1.23 to test new cookbooks, [[phab:T298005|T298005]] [[phab:T343869|T343869]]

=== 2023-08-29 ===
* 13:06 wm-bot2: deployed kubernetes component jobs-emailer ({{Gerrit|6f9c8cf}}) - cookbook ran by taavi@runko
* 13:03 wm-bot2: deployed kubernetes component jobs-api ({{Gerrit|b29193d}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-28 ===
* 14:54 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|90055b5}}) ([[phab:T344502|T344502]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-22 ===
* 14:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|27328a4}}) ([[phab:T344668|T344668]]) - cookbook ran by taavi@runko

=== 2023-08-18 ===
* 13:40 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|06c26be}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay
* 12:30 wm-bot2: deployed kubernetes component builds-api ({{Gerrit|727e6a7}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay

=== 2023-08-17 ===
* 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-{{Gerrit|25c2b55f}}

=== 2023-08-11 ===
* 09:06 taavi: fixed /etc/hosts on toolsbeta-nfs-2 because '{{fqdn}}' is not a valid fqdn

=== 2023-07-26 ===
* 09:30 wm-bot2: deployed kubernetes component image-config ({{Gerrit|06066ba}}) - cookbook ran by taavi@runko

=== 2023-07-25 ===
* 12:59 wm-bot2: deployed kubernetes component image-config ({{Gerrit|0eb287a}}) - cookbook ran by taavi@runko

=== 2023-07-20 ===
* 14:34 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image ([[phab:T342338|T342338]], [[phab:T321188|T321188]])
* 10:48 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 on toolsbeta

=== 2023-07-18 ===
* 10:45 arturo: redeploy jobs-emailer into k8s ([[phab:T341084|T341084]])

=== 2023-07-13 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|75db740}}) - cookbook ran by taavi@runko

=== 2023-07-12 ===
* 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-{{Gerrit|2ef80a7c}} ([[phab:T341084|T341084]])

=== 2023-07-04 ===
* 13:55 taavi: removed floating IP and public dns records for the harbor server

=== 2023-07-03 ===
* 19:08 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git ({{Gerrit|561b4d9}}) - cookbook ran by taavi@runko
* 08:57 wm-bot2: dcaro doing tests - cookbook ran by dcaro@urcuchillay

=== 2023-06-26 ===
* 07:49 dcaro: restarting harbor trove DB (in error status)

=== 2023-06-21 ===
* 11:48 dcaro: deploy bulids-api 0.2.0 ([[phab:T337025|T337025]])
* 11:48 dcaro: deploy bulids-api 0.2.0

=== 2023-06-16 ===
* 14:28 dcaro: deployed envvars-api 0.0.1
* 07:41 dcaro: deployed latest builds-api 0.1.0

=== 2023-06-15 ===
* 14:05 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by andrew@bullseye

=== 2023-06-08 ===
* 11:54 dcaro: powering off toolsbeta-test-k8s-etcd-22 ([[phab:T334644|T334644]])

=== 2023-06-07 ===
* 12:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ed420b}}) - cookbook ran by taavi@runko

=== 2023-06-01 ===
* 10:04 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|7e57832}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:11 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0f4076a}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 09:02 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|f1d94f7}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|6c6a27b}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus
* 07:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|3488cfe}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-26 ===
* 12:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|d567670}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-25 ===
* 08:40 dcaro: releasing toolforge-weld 1.0.0 ([[phab:T337218|T337218]])

=== 2023-05-24 ===
* 12:26 dcaro: deploy latest buildservice ([[phab:T335865|T335865]])
* 12:26 dcaro: deploy latest buildservice ([[phab:T336050|T336050]])

=== 2023-05-23 ===
* 14:40 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|0c7b25b}}) - cookbook ran by fran@wmf3169

=== 2023-05-16 ===
* 14:45 dcaro: deploy builds-api ([[phab:T336225|T336225]])
* 14:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|1a725d0}}) - cookbook ran by dcaro@vulcanus
* 11:45 dcaro: release toolforge-weld 0.2.0 and toolforge-webservice 0.98

=== 2023-05-15 ===
* 13:31 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0277378}}) - cookbook ran by dcaro@vulcanus
* 09:22 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller ({{Gerrit|ad5b2b5}}) - cookbook ran by dcaro@vulcanus

=== 2023-05-09 ===
* 17:05 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|e89c581}}) - cookbook ran by taavi@runko
* 07:27 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 07:24 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2023-05-05 ===
* 11:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|87937cd}}) - cookbook ran by taavi@runko

=== 2023-05-01 ===
* 23:24 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7199a9e}}) - cookbook ran by raymond@ubuntu

=== 2023-04-30 ===
* 14:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-19 - cookbook ran by taavi@runko
* 14:42 wm-bot2: removed instance toolsbeta-test-k8s-etcd-18 - cookbook ran by taavi@runko
* 14:33 wm-bot2: removed instance toolsbeta-test-k8s-etcd-17 - cookbook ran by taavi@runko

=== 2023-04-19 ===
* 16:17 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 14:29 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 14:09 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:45 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 13:34 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:32 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko
* 12:10 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko
* 12:07 wm-bot2: removed instance toolsbeta-test-k8s-etcd-22 - cookbook ran by taavi@runko

=== 2023-04-11 ===
* 14:13 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git ({{Gerrit|d878e49}}) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|b65439b}}) - cookbook ran by arturo@nostromo
* 10:27 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|8f0bfcd}}) - cookbook ran by taavi@runko
* 08:59 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko
* 08:46 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko
* 08:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico ({{Gerrit|c6a3e29}}) - cookbook ran by taavi@runko

=== 2023-04-05 ===
* 15:53 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 15:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|5ea5992}}) - cookbook ran by taavi@runko
* 15:12 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|2be9962}}) - cookbook ran by taavi@runko

=== 2023-04-03 ===
* 11:14 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 11:13 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 11:12 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 11:11 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-3 - cookbook ran by arturo@nostromo
* 11:10 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-4 - cookbook ran by arturo@nostromo
* 11:08 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-5 - cookbook ran by arturo@nostromo
* 11:07 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-6 - cookbook ran by arturo@nostromo
* 11:05 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 11:03 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-8 - cookbook ran by arturo@nostromo
* 11:01 wm-bot2: rebooting the whole toolsbeta k8s cluster (9 nodes) - cookbook ran by arturo@nostromo
* 11:00 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:59 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:26 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo
* 10:24 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 10:22 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo

=== 2023-03-19 ===
* 09:32 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by taavi@runko

=== 2023-03-14 ===
* 10:39 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b70adc1}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local
* 10:23 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7d4afeb}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local

=== 2023-03-13 ===
* 09:27 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-03-10 ===
* 16:35 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|8b42b15}}) - cookbook ran by taavi@runko

=== 2023-03-09 ===
* 10:08 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|53e7f81}}) - cookbook ran by taavi@runko

=== 2023-03-07 ===
* 11:09 taavi: upgrading kubernetes to 1.22 [[phab:T286856|T286856]]

=== 2023-03-06 ===
* 12:48 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|6688477}}) - cookbook ran by taavi@runko
* 12:45 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|21fef22}}) - cookbook ran by taavi@runko
* 12:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|98ce17f}}) - cookbook ran by taavi@runko
* 12:00 arturo: delete calico deployment, and try loading it again for https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1

=== 2023-03-05 ===
* 15:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|3e04025}}) - cookbook ran by taavi@runko

=== 2023-03-02 ===
* 11:31 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml (https://gerrit.wikimedia.org/r/c/operations/puppet/+/889836)

=== 2023-03-01 ===
* 13:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13eda9d}}) - cookbook ran by taavi@runko

=== 2023-02-28 ===
* 17:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|9252af7}}) - cookbook ran by taavi@runko
* 17:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e46da83}}) - cookbook ran by taavi@runko
* 14:11 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus

=== 2023-02-23 ===
* 16:37 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|efb60b3}}) - cookbook ran by taavi@runko
* 16:30 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|4e8645a}}) - cookbook ran by taavi@runko

=== 2023-02-17 ===
* 11:27 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|eeeea4c}}) - cookbook ran by arturo@endurance
* 11:17 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|7729b18}}) ([[phab:T254636|T254636]]) - cookbook ran by arturo@endurance

=== 2023-02-16 ===
* 16:01 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:55 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo
* 15:28 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager ({{Gerrit|d71994e}}) - cookbook ran by arturo@nostromo
* 13:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|7191997}}) - cookbook ran by taavi@runko
* 10:32 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml

=== 2023-02-15 ===
* 09:30 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by arturo@nostromo

=== 2023-02-14 ===
* 20:52 taavi: deploy cert-manager to toolsbeta [[phab:T329453|T329453]]
* 12:02 arturo: included tools-manifests 0.25 in toolsbeta-buster aptly repo ([[phab:T329611|T329611]], [[phab:T329467|T329467]], [[phab:T244809|T244809]])

=== 2023-02-13 ===
* 15:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13d87c4}}) - cookbook ran by taavi@runko
* 13:55 wm-bot2: drained, depooled and removed worker toolsbeta-test-k8s-worker-5 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Drained node toolsbeta-test-k8s-worker-4 - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by arturo@nostromo
* 13:45 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:31 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:30 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
* 13:15 arturo: cordoned &amp; drained k8s workers 4 to 7 to force workload to relocate to 8 ([[phab:T329378|T329378]])
* 12:35 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-8.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by arturo@nostromo
* 12:24 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-10 ===
* 16:14 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo

=== 2023-02-01 ===
* 15:41 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|372037f}}) - cookbook ran by taavi@runko

=== 2023-01-26 ===
* 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|307f302}}) - cookbook ran by taavi@runko

=== 2023-01-23 ===
* 11:26 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d5ae229}}) ([[phab:T311918|T311918]]) - cookbook ran by taavi@runko

=== 2023-01-20 ===
* 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo
* 15:56 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo
* 15:54 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo

=== 2023-01-19 ===
* 11:46 arturo: `aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff)

=== 2023-01-18 ===
* 15:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ad4c66}}) - cookbook ran by arturo@nostromo

=== 2023-01-17 ===
* 13:56 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8cf38a1}}) - cookbook ran by arturo@endurance
* 13:46 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0d0a882}}) - cookbook ran by arturo@endurance
* 13:45 arturo: add login.toolsbeta.wmflabs.org DNS record as CNAME to toolsbeta-sgebastion-05.toolsbeta.eqiad1.wikimedia.cloud

=== 2023-01-10 ===
* 11:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8e0a2f9}}) - cookbook ran by arturo@endurance
* 10:42 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0243967}}) - cookbook ran by arturo@endurance

=== 2022-12-09 ===
* 08:45 dcaro: manually started puppetdb after killed by oom ([[phab:T324812|T324812]])

=== 2022-11-30 ===
* 10:37 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|bc3529d}}) - cookbook ran by arturo@nostromo

=== 2022-11-29 ===
* 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|864171a}}) - cookbook ran by taavi@runko
* 12:22 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|a8b6e17}}) - cookbook ran by taavi@runko
* 09:54 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|9528ed3}}) - cookbook ran by taavi@runko

=== 2022-11-28 ===
* 18:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|ec5c82b}}) - cookbook ran by taavi@runko
* 18:36 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|5394a34}}) - cookbook ran by taavi@runko

=== 2022-11-15 ===
* 12:40 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 11:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu

=== 2022-11-14 ===
* 20:05 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 19:58 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu
* 14:14 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:14 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169
* 14:12 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169

=== 2022-11-07 ===
* 13:32 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b4e912e}}) - cookbook ran by fran@wmf3169

=== 2022-11-04 ===
* 12:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d464be4}}) ([[phab:T304900|T304900]]) - cookbook ran by arturo@nostromo

=== 2022-11-01 ===
* 12:42 taavi: remove labstore1006/7 from acme-chief-1 fstab and reboot

=== 2022-10-24 ===
* 16:42 wm-bot2: rebooted buster webgen grid workers - cookbook ran by andrew@bullseye
* 16:29 wm-bot2: rebooting buster webgen grid workers - cookbook ran by andrew@bullseye
* 14:54 wm-bot2: Increased quotas by 30 gigabytes - cookbook ran by dcaro@vulcanus

=== 2022-10-18 ===
* 10:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|64385e9}}) ([[phab:T320405|T320405]]) - cookbook ran by arturo@nostromo

=== 2022-10-17 ===
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-10-14 ===
* 07:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0cc020e}}) - cookbook ran by taavi@runko

=== 2022-10-12 ===
* 10:29 dcaro: deploying new registry-admission controller

=== 2022-10-10 ===
* 08:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|afa90ed}}) ([[phab:T320284|T320284]]) - cookbook ran by taavi@runko

=== 2022-09-28 ===
* 09:48 arturo: manually starting gridengine-master.service on toolsbeta-sgegrid-master ([[phab:T318788|T318788]])

=== 2022-09-27 ===
* 14:23 arturo: briefly livehacking puppetmaster

=== 2022-08-24 ===
* 11:55 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|7d0e951}}) - cookbook ran by taavi@runko

=== 2022-08-12 ===
* 07:24 dcaro_away: started postgresql on puppetdb-02, might have crashed during the ceph issues, now puppet runs on toolsbeta work again

=== 2022-08-03 ===
* 15:46 dhinus: recreated jobs-api pods to pick up new ConfigMap
* 14:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|c47ac41}}) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-01 ===
* 14:01 taavi: unbreak acme-chief after keystone communication issues

=== 2022-07-19 ===
* 15:45 taavi: deploying and testing maintain-kubeusers updates

=== 2022-06-28 ===
* 15:23 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko

=== 2022-06-24 ===
* 07:01 wm-bot2: removing grid node toolsbeta-sgewebgrid-lighttpd-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:59 wm-bot2: removing grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:57 wm-bot2: removing grid node toolsbeta-sgeexec-0902.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
* 06:55 wm-bot2: removing grid node toolsbeta-sgeexec-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

=== 2022-06-19 ===
* 16:28 taavi: restart OOM'd puppetdb on toolsbeta-puppetdb-02

=== 2022-06-03 ===
* 13:17 bd808: publish tools-webservice 0.86 ([[phab:T309821|T309821]])
* 05:25 wm-bot2: rebooted buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting buster weblight grid workers - cookbook ran by taavi@runko
* 05:20 wm-bot2: rebooting stretch weblight grid workers - cookbook ran by taavi@runko

=== 2022-05-30 ===
* 13:42 taavi: run grid-configurator to remove stale config for some removed nodes

=== 2022-05-26 ===
* 15:38 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e6fa299}}) - cookbook ran by taavi@runko

=== 2022-04-20 ===
* 07:53 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8f37a04}}) ([[phab:T305592|T305592]]) - cookbook ran by taavi@runko

=== 2022-04-15 ===
* 13:26 taavi: shutdown toolsbeta-services-01, not exactly sure what it does and it has no roles applied [[phab:T306100|T306100]]

=== 2022-04-11 ===
* 14:47 dcaro: deploying custom version of the regitsry admission hook

=== 2022-04-08 ===
* 10:45 arturo: disabled debug mode on the k8s jobs-emailer component

=== 2022-04-05 ===
* 07:43 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d7d3463}}) - cookbook ran by arturo@nostromo
* 07:21 arturo: deploying toolforge-jobs-framework-cli v7

=== 2022-04-04 ===
* 16:58 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|cbcfc47}}) - cookbook ran by arturo@nostromo
* 09:28 arturo: deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions

=== 2022-03-25 ===
* 11:31 dcaro: All alerting VMs rebooted, checking that everything is &quot;working&quot; ([[phab:T304672|T304672]])
* 10:55 dcaro: force restarting all the other nfs-bound VMs one by one ([[phab:T304672|T304672]])
* 10:43 dcaro: restarting the sge-shadow ([[phab:T304672|T304672]])
* 10:32 dcaro: restarting the sge-master ([[phab:T304672|T304672]])

=== 2022-03-16 ===
* 15:23 taavi: deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a [[phab:T292238|T292238]] test to toolsbeta

=== 2022-03-15 ===
* 17:55 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|084ee51}}) - cookbook ran by arturo@nostromo

=== 2022-03-14 ===
* 16:14 wm-bot: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus

=== 2022-03-11 ===
* 15:55 dcaro: added provisional toolforg cli package to toolsbeta buster repo ([[phab:T299026|T299026]])
* 15:11 dcaro: added tekton cli package to toolsbeta repos ([[phab:T299026|T299026]])
* 15:02 arturo: deploy jobs-framework-emailer {{Gerrit|9470a5f}} ([[phab:T286135|T286135]])
* 11:59 arturo: deploy jobs-framework-emailer {{Gerrit|d60ffd6}} ([[phab:T286135|T286135]])

=== 2022-03-08 ===
* 08:20 taavi: reboot toolsbeta-cumin-1 for kernel updates

=== 2022-03-07 ===
* 15:44 dcaro: Deployed buildpack-admission-controller with the latest code ([[phab:T297090|T297090]])

=== 2022-02-17 ===
* 08:16 taavi: made toolsbeta-puppetmaster-04 its own client to fix `puppet node deactivate` puppetdb access

=== 2022-02-08 ===
* 13:04 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/760933 ([[phab:T284767|T284767]])
* 12:19 arturo: created puppet prefix `toolsbeta-sgecron` with proper hiera/roles
* 12:16 arturo: created VM toolsbeta-sgecron-02 ([[phab:T284767|T284767]])

=== 2022-02-04 ===
* 18:53 taavi: upgrading to kubernetes 1.21 [[phab:T282942|T282942]]

=== 2022-01-28 ===
* 16:28 wm-bot: trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo

=== 2022-01-25 ===
* 11:45 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2022-01-20 ===
* 12:35 wm-bot: removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 12:34 wm-bot: removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-19 ===
* 14:11 arturo: craeted 'automated-toolforge-tests' tool account following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta

=== 2022-01-18 ===
* 15:56 wm-bot: removing grid node toolsbeta-sgewebgrid-generic-0901 (depool/drain, remove VM and reconfigure grid) - cookbook ran by andrew@buster
* 15:30 andrewbogott: switching scratch mount over to the cloud-hosted service with git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/43/754043/1 &amp;&amp; git cherry-pick FETCH_HEAD
* 09:46 arturo: creating VM toolsbeta-sgebastion-05, deleting toolsbeta-bastion-05 (wrong prefix)

=== 2022-01-17 ===
* 18:09 wm-bot: pooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@nostromo
* 18:07 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo
* 17:54 wm-bot: removing grid node toolsbeta-sgewebgen-10-4 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 13:39 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo

=== 2022-01-14 ===
* 11:56 wm-bot: removing grid node toolsbeta-sgewebgen-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 11:49 wm-bot: removing grid node toolsbeta-sgeexec-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:57 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:53 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.org (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
* 09:44 wm-bot: removing grid node toolsbeta-sgeweblight-10-2 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo

=== 2022-01-12 ===
* 12:28 wm-bot: created node toolsbeta-sgeweblight-10-1.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo
* 11:27 arturo: created puppet prefix `toolsbeta-sgeweblight`, drop `toolsbeta-sgeweblig`
* 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig'
* 11:00 wm-bot: created node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo

=== 2022-01-11 ===
* 11:11 wm-bot: created a grid exec node toolsbeta-sgeexec-10-5.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
* 09:20 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo

=== 2021-12-23 ===
* 13:32 wm-bot: trying to join node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 12:11 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-10-4.toolsbeta.eqiad1.wikimedia.cloud to the pool - cookbook ran by arturo@endurance
* 11:58 wm-bot: node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
* 11:40 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 11:26 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:25 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2 to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:24 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:59 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:34 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 10:31 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance

=== 2021-12-22 ===
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 12:01 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
* 11:24 wm-bot: removing instance toolsbeta-sgewebgen-09-1 - cookbook ran by arturo@endurance
* 11:21 wm-bot: removing grid node toolsbeta-sgewebgen-09-1 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@endurance
* 11:19 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance
* 10:42 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance

=== 2021-12-21 ===
* 16:32 wm-bot: removing instance toolsbeta-sgewebgen-10-2 - cookbook ran by arturo@endurance
* 16:24 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 16:24 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:50 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:07 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:04 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:04 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 12:03 wm-bot: Node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
* 12:03 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:48 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
* 11:06 arturo: bump quotas, instances from 50 to 55, CPU from 100 to 150, RAM from 200GB to 250GB ([[phab:T277653|T277653]])

=== 2021-12-16 ===
* 12:46 wm-bot: Joining grid node toolsbeta-sgewebgen-10-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance

=== 2021-12-15 ===
* 14:03 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:31 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
* 13:29 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance

=== 2021-12-08 ===
* 05:15 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1028

=== 2021-11-28 ===
* 17:44 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance

=== 2021-11-16 ===
* 12:37 majavah: testing calico 3.21 upgrade [[phab:T292698|T292698]]

=== 2021-11-05 ===
* 19:07 majavah: testing registry-admission changes

=== 2021-10-28 ===
* 12:48 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true`

=== 2021-10-25 ===
* 14:41 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image [[phab:T292771|T292771]]

=== 2021-10-20 ===
* 12:15 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories

=== 2021-10-16 ===
* 07:47 majavah: deployed cert-manager and wave as a test for automating [[phab:T292238|T292238]]

=== 2021-10-14 ===
* 15:02 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:01 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus

=== 2021-10-13 ===
* 11:18 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-12 ===
* 16:10 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:05 majavah: start gridengine-master.service on toolsbeta-sgegrid-master

=== 2021-10-11 ===
* 15:24 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:32 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-07 ===
* 14:21 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:55 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 08:04 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 07:58 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-06 ===
* 10:36 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:08 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus

=== 2021-10-04 ===
* 17:07 bstorm: reboot everything [[phab:T291406|T291406]]
* 17:06 bstorm: use cumin to edit fstab to remove old nfs mounts [[phab:T291406|T291406]]
* 16:41 bstorm: setting mount_nfs: true on toolsbeta-mail prefix (which is the correct setting)
* 14:45 dcaro: rebooting toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad1.wikimedia.cloud to force a fsck of the dm-0 device on boot ([[phab:T290970|T290970]])

=== 2021-10-01 ===
* 12:34 arturo: rebooting toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])
* 12:12 arturo: experimenting with newer mono runtime on toolsbeta-sgebastion-04 ([[phab:T292289|T292289]])

=== 2021-09-29 ===
* 22:13 bstorm: ran label fix script to use new label format
* 22:12 bstorm: toollabs-webservice 0.77 deployed

=== 2021-09-28 ===
* 10:32 majavah: removing all podpreset objects and disabling settings.k8s.io/v1alpha1 api

=== 2021-09-27 ===
* 16:13 majavah: testing volume-admission fix for containers with some volumes mounted

=== 2021-09-23 ===
* 17:14 majavah: testing new maintain-kubeusers release [[phab:T279106|T279106]]

=== 2021-09-22 ===
* 18:07 bstorm: launching toolsbeta-nfs-test-client-01 to run a &quot;fair&quot; test battery against [[phab:T291406|T291406]]

=== 2021-09-15 ===
* 08:04 majavah: tools-manifest 0.24, [[phab:T290325|T290325]]

=== 2021-09-14 ===
* 15:45 majavah: disable podpreset admission plugin in toolsbeta [[phab:T279106|T279106]]
* 11:42 arturo: deploying jobs-framework-emailer {{Gerrit|3045601}} ([[phab:T286135|T286135]])
* 10:44 arturo: deploying jobs-framework-emailer {{Gerrit|51032af}} ([[phab:T286135|T286135]])
* 10:39 arturo: deploying jobs-framework-api {{Gerrit|16fbf51}} ([[phab:T286135|T286135]])

=== 2021-09-13 ===
* 15:44 majavah: deploy volume-admission-controller in background; [[phab:T279106|T279106]]

=== 2021-09-09 ===
* 17:36 bstorm: deploying a base tekton triggers setup [[phab:T267374|T267374]]
* 16:50 majavah: enable unattended updates on toolsbeta [[phab:T290494|T290494]]
* 16:19 arturo: {{Gerrit|70017ec0ac}} root@toolsbeta-test-k8s-control-4:~# kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml
* 00:26 bstorm: deleted toolsbeta-sgeexec-0902 since it had a badly screwed up /tmp

=== 2021-09-03 ===
* 22:34 bstorm: backfilled quotas for [[phab:T286784|T286784]]

=== 2021-08-30 ===
* 23:23 bstorm: deleting toolsbeta-workflow-test [[phab:T289709|T289709]]

=== 2021-08-21 ===
* 00:17 bstorm: rebooting the control plane nodes for kubernetes because it can't make things worse [[phab:T289390|T289390]]

=== 2021-08-20 ===
* 23:19 bstorm: tried renewing all the certs to get certs working again in kubernetes

=== 2021-08-12 ===
* 16:55 bstorm: deployed updated manifest for ingress-admission
* 15:02 majavah: deploying ingress-admission-controller using v1 api [[phab:T280436|T280436]]

=== 2021-07-30 ===
* 08:01 majavah: replace toolsbeta-sgeexec-1002 with -1004 for [[phab:T287666|T287666]]

=== 2021-07-29 ===
* 14:08 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]]
* 13:06 majavah: rebuild toolsbeta-sgeexec-1001 as -1003 [[phab:T287666|T287666]]

=== 2021-07-23 ===
* 13:31 majavah: upgrading toolsbeta to kubernetes 1.19, [[phab:T280340|T280340]]

=== 2021-07-22 ===
* 15:32 arturo: re-deploying toolforge-jobs-framework-api

=== 2021-07-21 ===
* 11:58 arturo: deploying jobs-framework-api {{Gerrit|07346d715d17585db9c16dd152cc91ef0bea33c3}} ([[phab:T286108|T286108]])
* 10:51 arturo: enabling TTLAfterFinished feature gate on static pod manifests on /etc/kubernetes/manifests/kube-&lt;nowiki&gt;{&lt;/nowiki&gt;apiserver,controller-manager&lt;nowiki&gt;}&lt;/nowiki&gt;.yaml in all 3 control nodes ([[phab:T286108|T286108]])
* 10:47 arturo: enabling TTLAfterFinished feature gate on kubeadm live configmap ([[phab:T286108|T286108]])
* 10:09 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/705848

=== 2021-07-20 ===
* 21:18 bstorm: applied `login_server: true` to toolsbeta-sgecron-01 [[phab:T287037|T287037]]
* 19:09 bstorm: upgraded version of maintain-kubeusers to the latest in master branch [[phab:T285011|T285011]]
* 08:36 majavah: resolve merge conflicts on labs/private

=== 2021-07-16 ===
* 19:53 bstorm: set matchPolicy to equivalent on ingress admission controller for toolsbeta [[phab:T280360|T280360]]
* 14:04 arturo: deployed jobs-framework-api {{Gerrit|42b7a88}} ([[phab:T286132|T286132]])

=== 2021-07-15 ===
* 15:39 arturo: deploy toolforge-jobs-framework-api git version {{Gerrit|d85d93ee1c5d4be6a526cf83e806b2679dde3875}}

=== 2021-07-14 ===
* 09:05 majavah: testing calico 3.18 upgrade - [[phab:T280342|T280342]]

=== 2021-07-12 ===
* 11:42 majavah: rebooting toolsbeta-sgeexec-1002, nfs issues

=== 2021-07-07 ===
* 09:48 majavah: set dummy values for openstack ldap user/pass hiera values for disable_tool manifests to work

=== 2021-07-01 ===
* 17:01 majavah: updating jobs-framework-api
* 10:00 arturo: refreshed jobs-api deployment

=== 2021-06-29 ===
* 09:28 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:28 wm-bot: Drained node toolsbeta-test-k8s-worker-3. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Draining node toolsbeta-test-k8s-worker-3... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:27 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-6.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-2.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Drained node toolsbeta-test-k8s-worker-2. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Draining node toolsbeta-test-k8s-worker-2... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 09:09 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-5.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-1.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot: Drained node toolsbeta-test-k8s-worker-1. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:58 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus
* 08:57 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-28 ===
* 14:46 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Drained node toolsbeta-test-k8s-worker-4. - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Depooling and removing worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:22 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus
* 13:16 wm-bot: Draining node toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud... - cookbook ran by dcaro@vulcanus
* 11:30 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:25 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:23 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:12 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:54 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:53 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 10:11 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 09:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:51 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 08:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-25 ===
* 15:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:17 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:02 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:45 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 13:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-24 ===
* 15:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:33 dcaro: created flavor g3.cores4.ram8.disk20.ephem40 for the k8s workers
* 15:10 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 15:09 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:28 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus

=== 2021-06-22 ===
* 18:24 majavah: rolling out kubernetes patch release 1.18.20, cluster is currently at 1.18.18

=== 2021-06-17 ===
* 11:44 majavah: toolsbeta-puppetdb-02: stop puppetdb to free up its ram usage, start postgres process, start puppetdb up again

=== 2021-06-16 ===
* 15:53 majavah: add default security group rule allowing prometheus01.metricsinfra to connect to node-exporter port 9100

=== 2021-06-15 ===
* 16:10 majavah: set toolsbeta-bastion-05 as grid submit host

=== 2021-06-14 ===
* 21:29 bstorm: deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 [[phab:T282975|T282975]]
* 10:19 arturo: deploying toolforge jobs-framework-api in kubernetes (just a test) ([[phab:T283238|T283238]])

=== 2021-06-12 ===
* 14:42 majavah: sync hiera key prometheus_nodes to match tools

=== 2021-06-11 ===
* 15:25 majavah: undeploy nginx-ingress-jobs from kubernetes
* 14:54 majavah: generate and add own root key to passwords::root::extra_keys

=== 2021-06-08 ===
* 15:11 majavah: updating k8s worker nodes to 1.18 [[phab:T280299|T280299]]
* 15:02 majavah: continuing to update k8s ingress nodes [[phab:T280299|T280299]]
* 14:57 majavah: continuing to update rest of k8s control nodes [[phab:T280299|T280299]]
* 14:42 majavah: remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work
* 14:08 majavah: update toolsbeta-test-k8s-control-4 to kubernetes 1.18 [[phab:T280299|T280299]]

=== 2021-06-03 ===
* 16:55 majavah: renew ingress-admission-controller certificates [[phab:T280301|T280301]]
* 16:49 majavah: renew registry-admission-webhook certificates [[phab:T280301|T280301]]

=== 2021-05-25 ===
* 17:14 andrewbogott: deleting old ingress controllers toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2
* 17:13 andrewbogott: created two new ingress nodes, toolsbeta-test-k8s-ingress-4 and toolsbeta-test-k8s-ingress-5
* 15:09 dcaro: turning off VM toolsbeta-test-k8s-etcd-14 to be able to reboot cloudvirt1020

=== 2021-05-24 ===
* 19:40 andrewbogott: replacing existing etcd nodes with localdisk nodes

=== 2021-05-19 ===
* 11:35 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/692875/
* 06:51 Majavah: depool toolsbeta-test-k8s-ingress-1

=== 2021-05-15 ===
* 07:52 Majavah: set profile::wmcs::kubeadm::control::apiserver_cert_alternative_names hiera key and adjust config map [[phab:T262562|T262562]]

=== 2021-05-14 ===
* 11:22 arturo: allowed VIP address from the new port 172.16.3.26 into the ports of toolsbeta-redis-[1-3] ([[phab:T153810|T153810]])
* 11:16 arturo: aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-redis-vip ([[phab:T153810|T153810]])

=== 2021-05-13 ===
* 08:07 Majavah: creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / [[phab:T153810|T153810]]

=== 2021-05-10 ===
* 19:42 bstorm: setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes
* 17:43 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta [[phab:T264221|T264221]]
* 11:50 Majavah: testing ingress-nginx update https://gerrit.wikimedia.org/r/c/operations/puppet/+/685715 on toolsbeta [[phab:T264221|T264221]]

=== 2021-05-08 ===
* 10:42 Majavah: create new ingress node toolsbeta-k8s-ingress-3 [[phab:T264221|T264221]]

=== 2021-05-07 ===
* 17:00 bstorm: deleted &quot;toolsbeta-test-k8s-haproxy-2&quot;, &quot;toolsbeta-test-k8s-haproxy-1&quot; when the dns caches finally dropped [[phab:T282227|T282227]]
* 16:30 bstorm: recreated k8s.toolsbeta.eqiad1.wikimedia.cloud. as a CNAME to k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. [[phab:T282227|T282227]]
* 16:16 Majavah: create record k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. pointing to haproxy vip [[phab:T282227|T282227]]
* 14:20 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/686607/
* 09:44 arturo: `sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-k8s-haproxy-keepalived-vip`
* 08:19 Majavah: rebuild toolsbeta-test-k8s-haproxy-[12] without nfs

=== 2021-05-05 ===
* 16:25 Majavah: add self to sudo policy `roots`
* 16:07 arturo: grant `taavi` projectadmin (Majavah)

=== 2021-05-04 ===
* 10:47 arturo: rebase &amp; resolve merge conflicts in labs/private.git

=== 2021-05-03 ===
* 13:23 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 ([[phab:T278109|T278109]])

=== 2021-04-29 ===
* 18:10 bstorm: added and removed an etcd node

=== 2021-04-23 ===
* 17:24 bstorm: rebooting toolsbeta-test-k8s-control-6 because it was &quot;notready&quot; for some reason

=== 2021-04-20 ===
* 19:01 bstorm: updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244

=== 2021-04-13 ===
* 16:41 arturo: create VM toolsbeta-sgeexec-1002 ([[phab:T277653|T277653]])
* 15:44 arturo: delete VMs toolsbeta-sgeexec-0903 and toolsbeta-buster-sgeexec-01 (no longer useful)
* 15:36 arturo: created VM toolsbeta-sgeexec-0903 (buster) ([[phab:T277653|T277653]])
* 15:31 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/678043/ ([[phab:T277653|T277653]])

=== 2021-04-08 ===
* 18:27 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns [[phab:T277653|T277653]]

=== 2021-04-06 ===
* 13:11 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number  ([[phab:T267082|T267082]])

=== 2021-04-01 ===
* 15:17 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook)
* 14:54 dcaro: shrinking etcd cluster to 3 members, cleaning up automation runs

=== 2021-03-31 ===
* 18:22 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo [[phab:T275478|T275478]]

=== 2021-03-24 ===
* 12:17 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM
* 11:41 arturo: created VM toolsbeta-docker-registry-02 as Debian buster ([[phab:T278303|T278303]])
* 11:34 arturo: attached cinder volume `toolsbeta-docker-registry-data` as /dev/vdb on toolsbeta-docker-registry-01
* 11:23 arturo: created 2G cinder volume `toolsbeta-docker-registry-data` ([[phab:T278303|T278303]])

=== 2021-03-23 ===
* 11:22 arturo: drop and build again the VM toolsbeta-sgregrid-master ([[phab:T277653|T277653]])
* 11:07 arturo: drop and build again the VM toolsbeta-sgregrid-shadow ([[phab:T277653|T277653]])

=== 2021-03-18 ===
* 18:55 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix
* 18:50 arturo: deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project)
* 18:49 arturo: deleting VM toolsbeta-workflow-test, no longer useful
* 18:44 arturo: replacing toolsbeta-sgegrid-master with a Debian Buster VM ([[phab:T277653|T277653]])
* 16:24 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/672456
* 12:53 arturo: create anti-affinity server group toolsbeta-sgegrid-master-shadow
* 12:51 arturo: rebuild toolsbeta-sgegrid-shadow instance as debian buster ([[phab:T277653|T277653]])
* 12:50 arturo: added puppet prefix `toolsbeta-sgegrid-shadow`, migrate puppet config from VM to here
* 12:48 arturo: destroy VM toolsbeta-buster-gridmaster (no longer useful) [[phab:T277653|T277653]]
* 12:47 arturo: delete puppet prefix `toolsbeta-buster-grirdmaster` (no longer useful) [[phab:T277653|T277653]]

=== 2021-03-17 ===
* 12:39 arturo: created VM toolsbeta-buster-gridmaster ([[phab:T277653|T277653]])
* 12:38 arturo: created puppet prefix 'toolsbeta-buster-gridmaster' ([[phab:T277653|T277653]])
* 12:00 arturo: create VM toolsbeta-buster-sgeexec-01 ([[phab:T277653|T277653]])
* 11:56 arturo: created puppet prefix 'toolsbeta-buster-sgeexec' ([[phab:T277653|T277653]])
* 10:34 arturo: re-create toolsbeta-bastion-05 ([[phab:T275865|T275865]])

=== 2021-03-16 ===
* 12:32 arturo: added packages jobutils / misctools v1.41 to &lt;nowiki&gt;{&lt;/nowiki&gt;stretch,buster&lt;nowiki&gt;}&lt;/nowiki&gt;-toolsbeta aptly repository in tools-sge-services-03

=== 2021-03-11 ===
* 12:33 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/667144 for [[phab:T275865|T275865]]

=== 2021-03-10 ===
* 16:48 arturo: briefly stopping VM toolsbeta-test-k8s-etcd-8 to migrate hypervisor

=== 2021-02-26 ===
* 20:39 andrewbogott: rebooting all hosts
* 15:35 dcaro: removed toolsbeta-test-k8s-etcd-9 with depool from kubeadmin/etcd ([[phab:T274497|T274497]])
* 11:46 arturo: `openstack server create --os-project-id toolsbeta --image debian-10.0-buster --flavor g2.cores2.ram4.disk40 --network lan-flat-cloudinstances2b --property description='buster bastion test' toolsbeta-bastion-05` ([[phab:T275865|T275865]])
* 11:39 arturo: created puppet prefix 'toolsbeta-bastion' to hold new configuration for buster-based bastions ([[phab:T275865|T275865]])
* 09:09 dcaro: Playing around with cookbooks by adding/removing etcd nodes, etcd might missbehave from time to time ([[phab:T274497|T274497]])

=== 2021-02-19 ===
* 12:42 arturo: deploying new version of the ingress admission controller
* 11:46 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) which should only affect toolsbeta
* 10:27 arturo: create DNS record `jobs.svc.toolsbeta.eqiad1.wikimedia.cloud` with CNAME to `k8s.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])
* 10:25 arturo: create DNS zone `svc.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]])

=== 2021-02-10 ===
* 12:34 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]])
* 12:23 arturo: add `webserver` security group to toolsbeta-proxy-3 and -4
* 12:20 arturo: fix A record for `toolsbeta.wmflabs.org`, point it to 172.16.1.150  (toolsbeta-proxy-3), it was previously pointing to an old IP address

=== 2021-02-08 ===
* 11:48 arturo: trying to introduce TLS support in the front proxy [[phab:T274123|T274123]]

=== 2021-02-05 ===
* 00:36 bstorm: updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing

=== 2021-01-21 ===
* 15:29 bstorm: pushed the maintain-kubeusers:beta tag with the new code to the docker repo [[phab:T271847|T271847]]

=== 2021-01-13 ===
* 14:10 dcaro: dcaro doing puppet tests, puppet runs might break
* 10:07 arturo: allocate floating IP 185.15.56.84, and use it for docker-registry.toolsbeta.wmflabs.org (instance toolsbeta-docker-registry-01) ([[phab:T271867|T271867]])
* 10:05 arturo: release and delete floating IP 185.15.56.242 (docker-registry.toolsbeta.wmflabs.org) ([[phab:T271867|T271867]])

=== 2020-12-22 ===
* 10:48 arturo: rebase &amp; resolve ugly git merge conflict in labs/private.git

=== 2020-12-18 ===
* 10:52 arturo: live-hacking local puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/650470 ([[phab:T267966|T267966]])

=== 2020-12-14 ===
* 19:27 bstorm: create temporary instance toolsbeta-test-io-unthrottled [[phab:T267966|T267966]]
* 19:25 bstorm: created temporary instance toolsbeta-io-test-local  [[phab:T267966|T267966]]

=== 2020-12-11 ===
* 23:31 bstorm: increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts

=== 2020-12-10 ===
* 08:58 dcaro: starting a new etcd instance completely from ansible playbook (etcd-8) ([[phab:T267412|T267412]])

=== 2020-12-09 ===
* 15:30 dcaro: Playing aronud adding a new etcd node (k8s-etcd-7) ([[phab:T267412|T267412]])

=== 2020-12-04 ===
* 11:17 dcaro: Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity ([[phab:T267412|T267412]])
* 10:12 dcaro: Trying to create a whole new etcd member from ansible ([[phab:T267412|T267412]])

=== 2020-11-23 ===
* 14:17 dcaro: All control nodes re-imaged ([[phab:T267140|T267140]])
* 14:08 dcaro: Taking control-3 node out as control-6 is up and running ([[phab:T267140|T267140]])
* 11:12 dcaro: Launching control-6, to replace control-3 ([[phab:T267140|T267140]])
* 10:45 dcaro: Taking out control-2 node, replaced by control-5 (I saw one 503 reply on the proxy when creating control-5, fyi) ([[phab:T267140|T267140]])
* 10:32 dcaro: Creating new control-5 node (will replace control-2) ([[phab:T267140|T267140]])
* 09:58 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267140|T267140]])
* 09:57 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267195|T267195]])

=== 2020-11-18 ===
* 11:46 dcaro_: Modifying the security groupts to mirror tools ([[phab:T267140|T267140]])
* 10:50 dcaro_: Adding new control-4 node to the control cluster ([[phab:T267140|T267140]])

=== 2020-11-17 ===
* 15:32 dcaro: Creating new toolsbeta-test-k8s-control-4 node and adding it to the cluster ([[phab:T267140|T267140]])
* 12:09 Lucas_WMDE: &lt;dcaro&gt; 11:59:36 UTC – toolbeta up and running again, documented on the live doc for now, apsrever had the wrong config ([[phab:T267140|T267140]])
* 10:40 arturo: hand-edited /etc/kubernetes/manifests/kube-apiserver.yaml in all 3 k8s control nodes to account for new etcd servers ([[phab:T267140|T267140]])
* 08:58 dcaro: etcd hosts reimaged ([[phab:T267140|T267140]])
* 08:54 dcaro: etcd-4,5 and 6 are up and running, removing 1,2 and 3 ([[phab:T267140|T267140]])

=== 2020-11-16 ===
* 11:44 dcaro: etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster ([[phab:T267140|T267140]])
* 11:27 dcaro: Creating instance toolsbeta-test-k8s-etcd5 and adding to the etcd cluster ([[phab:T267140|T267140]])

=== 2020-11-10 ===
* 19:42 bstorm: safelisted &quot;argocd&quot; namespace with namespaceSelector for registry-admission controller
* 18:49 legoktm: associated floating IP to toolsbeta-docker-registry-01 and pointed DNS docker-registry.toolsbeta.wmflabs.org. at it
* 18:27 legoktm: creating toolsbeta-docker-imagebuilder-01 ([[phab:T267616|T267616]])
* 17:18 dcaro: launching instance toolsbeta-test-k8s-etcd-4 ([[phab:T267140|T267140]])
* 17:15 dcaro: removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) ([[phab:T267140|T267140]])
* 14:44 dcaro: taking down one of the test-k8s etcd nodes to reimage ([[phab:T267140|T267140]])

=== 2020-11-06 ===
* 23:44 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 [[phab:T263284|T263284]]
* 21:23 bstorm: upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 [[phab:T263284|T263284]]
* 15:56 dcaro: Deleting instances proxy-1 and proxy-2, that will finish the proxy rebuild ([[phab:T267140|T267140]])
* 15:53 dcaro: Removing proxy-1 and proxy-3 from hiera, proxy-3 stays as active and proxy-4 as backup ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave ([[phab:T267140|T267140]])
* 13:18 dcaro: bringin up a new proxy-4 instance as slave

=== 2020-11-05 ===
* 16:40 dcaro: Moving active proxy from proxy-1 to proxy-3 ([[phab:T267140|T267140]])
* 15:54 dcaro: Adding toolsbeta-proxy-3 to the list of slave proxies in hiera ([[phab:T267140|T267140]])

=== 2020-11-04 ===
* 15:42 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try ([[phab:T267140|T267140]])
* 15:21 dcaro: creating new proxy instance toolsbeta-proxy-03
* 15:18 arturo: dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use
* 15:16 arturo: dropping project hiera config for `toollabs::proxy::proxies`, no longer in use
* 11:46 dcaro: The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix

=== 2020-11-03 ===
* 16:04 arturo: add dcaro to the toolsbeta.admin LDAP group ([[phab:T266068|T266068]])
* 15:30 dcaro: [[phab:T267121|T267121]]: Puppetmaster replaced, also removed old puppetdb master from hiera, testing
* 15:07 dcaro: Replacing old puppetmaster 02 and 03 from hiera with 04
* 10:55 dcaro: dcaro investigating puppet errors on toolsbeta-puppetdb-02

=== 2020-11-02 ===
* 13:35 arturo: added dcaro as projectadmin &amp; user ([[phab:T266068|T266068]])

=== 2020-10-29 ===
* 22:20 legoktm: switched test tool over to use buildpack image ([[phab:T265681|T265681]])

=== 2020-10-28 ===
* 18:58 andrewbogott: deleting  toolsbeta-puppetmaster-03 — seems broken and unused

=== 2020-10-22 ===
* 16:22 bstorm: created buildpack psp for [[phab:T265557|T265557]]

=== 2020-09-10 ===
* 09:17 arturo: force-rebooting toolsbeta-test-haproxy-2 (unresponsive)
* 09:15 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/626133 ([[phab:T250172|T250172]])
* 09:00 arturo: tainted/labeld toolsbeta-test-k8s-ingress-1 (and -2) in the k8s cluster ([[phab:T250172|T250172]])
* 08:59 arturo: added toolsbeta-test-k8s-ingress-1 (and -2) to the k8s cluster ([[phab:T250172|T250172]])

=== 2020-09-09 ===
* 11:50 arturo: after force-rebooting everything, the k8s cluster seems to have recovered itself. magic.
* 11:45 arturo: force-rebooting the 3 k8s etcd nodes. They seem down
* 11:42 arturo: actually, the whole k8s cluster seems down? the API seems down at least
* 11:39 arturo: all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them
* 11:27 arturo: created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 ([[phab:T250172|T250172]])
* 11:25 arturo: created new server group toolsbeta-k8s-ingress ([[phab:T250172|T250172]])
* 11:24 arturo: created new puppet prefix `toolsbeta-test-k8s-ingress` ([[phab:T250172|T250172]])

=== 2020-07-15 ===
* 21:35 bstorm: set all of toolsbeta to mount NFS 4.2 except the bastion [[phab:T257945|T257945]]

=== 2020-07-14 ===
* 22:28 bstorm: rebooting toolsbeta-sgebastion-04 during NFS testing thing

=== 2020-07-08 ===
* 11:08 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 ([[phab:T234617|T234617]])

=== 2020-06-26 ===
* 12:12 arturo: puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 ([[phab:T120210|T120210]])

=== 2020-06-24 ===
* 12:55 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 ([[phab:T120225|T120225]])
* 12:23 arturo: live-hacking puppetmaster with exim prometheus stuff ([[phab:T175964|T175964]])
* 11:31 arturo: live-hack the puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607320 ([[phab:T175964|T175964]])
* 11:26 arturo: add TXT record `&quot;v=spf1 mx -all&quot;` [[phab:T120225|T120225]]
* 11:24 arturo: fix MX record for toolsbeta.wmflabs.org (missing trailing dot) [[phab:T120225|T120225]]

=== 2020-06-23 ===
* 13:10 arturo: added herron to the test tool for email testing
* 11:36 arturo: removing `benapetr` and adding myself to the test tool
* 11:02 arturo: setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix
* 10:55 arturo: allow ingress smtp/smtps traffic in the MTA security group
* 10:52 arturo: created MX record pointing to mail.toolsbeta.wmflabs.org
* 09:43 arturo: restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert
* 09:38 arturo: live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251

=== 2020-06-16 ===
* 22:54 bd808: Building webservice 0.72

=== 2020-06-15 ===
* 21:54 bstorm_: removed killgridjobs.sh from toolsbeta bastion [[phab:T157792|T157792]]
* 17:52 bd808: Building webservice 0.71

=== 2020-06-12 ===
* 19:41 bstorm_: set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test [[phab:T127559|T127559]]

=== 2020-06-11 ===
* 12:42 arturo: introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there
* 12:39 arturo: for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working
* 12:35 arturo: according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O&lt;nowiki&gt;{&lt;/nowiki&gt;project:toolsbeta&lt;nowiki&gt;}&lt;/nowiki&gt;' 'run-puppet-agent'` we are mostly back in business
* 12:14 arturo: try switching all VMs to toolsbeta-puppetmaster-04
* 12:14 arturo: poweroff toolsbeta-puppetmaster-03
* 12:12 arturo: copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04
* 11:53 arturo: create VM toolsbeta-puppetmaster-04
* 11:35 arturo: try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults
* 11:33 arturo: reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems
* 11:32 arturo: apparently every python script segfaults in toolsbeta-puppetmaster-03
* 11:27 arturo: puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03
* 11:21 arturo: puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb`

=== 2020-06-04 ===
* 21:06 andrewbogott: added krenair to toolsbeta.admin group in ldap

=== 2020-05-28 ===
* 11:27 arturo: cleanup livehackings
* 10:31 arturo: livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 ([[phab:T253816|T253816]])
* 10:30 arturo: livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139

=== 2020-05-27 ===
* 12:02 arturo: the k8s cluster is now running v1.16.10 ([[phab:T246122|T246122]])
* 11:05 arturo: trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` ([[phab:T246122|T246122]])
* 11:02 arturo: upgraded the rest of the k8s control plane nodes to 1.16.10 ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo ([[phab:T246122|T246122]])
* 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! ([[phab:T246122|T246122]])

=== 2020-05-26 ===
* 16:17 bstorm_: fix incorrect volume name in kubeadm-config [[phab:T246122|T246122]]
* 15:02 arturo: first k8s upgrade failed for yet-to-be-known reasons ([[phab:T246122|T246122]])
* 14:54 arturo: `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` ([[phab:T246122|T246122]])
* 14:54 arturo: bump installed version of kubeadm and kubectl to 1.16.10 ([[phab:T246122|T246122]])
* 09:57 arturo: installing kubectl/kubeadm 1.16.9 on k8s worker nodes ([[phab:T246122|T246122]])
* 09:56 arturo: installing kubectl/kubeadm 1.16.9 on k8s control nodes ([[phab:T246122|T246122]])
* 09:30 arturo: set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying [[phab:T246122|T246122]]
* 09:25 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again

=== 2020-05-21 ===
* 22:14 bd808: Building tools-webservice 0.70 via wmcs-package-build.py

=== 2020-05-19 ===
* 12:20 arturo: trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 ([[phab:T247422|T247422]])
* 10:18 arturo: `aborrero@toolsbeta-puppetdb-02:~$ sudo systemctl restart puppetdb`

=== 2020-05-15 ===
* 20:48 bstorm_: found an error in the new version of maintain-kubeusers, removing the deployment for now [[phab:T246059|T246059]]
* 20:35 bstorm_: updating the maintain-kubeusers image to be able to control admin accounts

=== 2020-05-14 ===
* 12:09 arturo: created puppet prefix toolsbeta-acme-chief in horizon ([[phab:T252762|T252762]])
* 12:08 arturo: created toolsbeta-acme-chief-01 VM ([[phab:T252762|T252762]])

=== 2020-05-12 ===
* 18:35 bstorm_: upgraded to using typha and rolled back to not doing so -- no affect on existing network [[phab:T250863|T250863]]
* 17:44 bstorm_: set the calico version to v3.14.0 because the new liveness probe isn't compatible with the old version. [[phab:T250863|T250863]]
* 17:36 bstorm_: deployed an updated bit of yaml for calico without upgrading the version first [[phab:T250863|T250863]]

=== 2020-05-08 ===
* 12:48 arturo: allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` ([[phab:T120225|T120225]])
* 12:24 arturo: added puppet prefix `toolsbeta-email` ([[phab:T120225|T120225]])

=== 2020-05-07 ===
* 16:33 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])
* 12:36 arturo: cleanup livehacks in toolsbeta-puppetmaster-03
* 11:12 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]])

=== 2020-05-06 ===
* 19:11 bstorm_: updated toollabs-webservice to 0.69 for toolsbeta
* 09:58 arturo: livehacking toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594471 ([[phab:T251297|T251297]])

=== 2020-05-05 ===
* 10:04 arturo: add herron as user and projectadmin, we will work on the email setup ([[phab:T120225|T120225]])
* 09:59 arturo: created VM toolsbeta-mail-01 ([[phab:T120225|T120225]])

=== 2020-05-04 ===
* 13:02 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project

=== 2020-04-29 ===
* 19:48 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta [[phab:T247455|T247455]]

=== 2020-04-20 ===
* 14:47 arturo: added joakino to toolsbeta.admin LDAP group
* 12:06 arturo: installing tools-webservice v0.68 for testing
* 11:05 arturo: poweroff `toolsbeta-services-01`. I suspect this VM is not in use because no puppet role is in used there
* 10:58 arturo: run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` the service was in failed state, causing puppet failures across the whole project

=== 2020-04-10 ===
* 19:32 bstorm_: deployed webservice 0.67 [[phab:T249843|T249843]]
* 18:59 bstorm_: delete toolsbeta-gitlab-01 and build toolsbeta-workflow-test [[phab:T249946|T249946]]
* 00:40 bd808: REbooting toolsbeta-sgebastion-04. NFS seemed messed up

=== 2020-04-08 ===
* 01:10 bstorm_: upgrade toollabs-webservice to 0.66 for qa [[phab:T249390|T249390]]

=== 2020-03-31 ===
* 23:39 bstorm_: deployed toollabs-webservice-0.65 to toolsbeta

=== 2020-03-30 ===
* 10:35 arturo: remove local changes in the puppet tree in toolsbeta-puppetmaster-03 (docker mount point)
* 10:30 arturo: remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use

=== 2020-03-24 ===
* 18:45 jeh: cleanup and remove toolsbeta-elastic7-[1,2,3] VMs (re-configuring hypervisor for local storage) [[phab:T243327|T243327]]

=== 2020-03-19 ===
* 23:18 Krenair: Shut down toolsbeta-puppet(db-01{{!}}master-02) - [[phab:T241719|T241719]]
* 19:20 arturo: live-hacking toolsbeta-proxy-1 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/579952 ([[phab:T234617|T234617]])

=== 2020-03-16 ===
* 21:38 bstorm_: removed lots of hiera related to the legacy k8s cluster [[phab:T246689|T246689]]
* 19:45 bstorm_: deleting toolsbeta-worker-1001, toolsbeta-k8s-master, toolsbeta-flannel-etcd-01 and  toolsbeta-k8s-etcd-01 [[phab:T246689|T246689]]
* 19:07 bstorm_: shutting down toolsbeta-flannel-etcd-01 [[phab:T246689|T246689]]
* 19:06 bstorm_: shutting down toolsbeta-worker-1001, toolsbeta-k8s-master and toolsbeta-k8s-etcd [[phab:T246689|T246689]]
* 14:37 arturo: live-hacking the toollabs-webservice package in toolsbeta-sgewebgrid-lighttpd-0901 as well
* 14:22 arturo: live-hacking the toollabs-webservice package in toolsbeta*-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 14:22 arturo: live-hacking the toollabs-webservice package in tools-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]])
* 13:49 arturo: deleting 50 jobs of the `test` tool in the grid to leave room for other tests
* 13:18 arturo: live-hack toolsbeta-puppetmaster-02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 ([[phab:T234617|T234617]])

=== 2020-03-11 ===
* 21:32 bstorm_: deployed jobutils_1.39 and miscutils_1.39 to toolsbeta

=== 2020-03-09 ===
* 13:11 arturo: created VM `toolsbeta-legacy-redirector` ([[phab:T247236|T247236]])
* 13:08 arturo: instance quota was full, bump it from 35 to 40

=== 2020-03-06 ===
* 16:22 bstorm_: updating maintain-kubeusers image to filter invalid tool names

=== 2020-03-05 ===
* 21:22 bstorm_: updated maintain-kubeusers to the latest version for toolsbeta only to live test

=== 2020-02-27 ===
* 19:19 bstorm_: upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing
* 16:03 jeh: create 3 new VMs toolsbeta-elastic7-0[1,2,3]
* 16:00 jeh: increase CloudVPS quota instance count for new elasticsearch servers

=== 2020-02-26 ===
* 20:35 bstorm_: hard rebooting the grid master for toolsbeta
* 20:20 jeh: restart toolsbeta-sgegrid-shadow

=== 2020-02-18 ===
* 23:20 bstorm_: added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists

=== 2020-02-10 ===
* 21:19 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.62 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-02-07 ===
* 23:07 bstorm_: upgraded toollabs-webservice for stetch toolsbeta to 0.60 [[phab:T244611|T244611]]
* 21:09 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.59 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]]

=== 2020-01-23 ===
* 03:14 bd808: Demoted projectadmins not listed in the &quot;roots&quot; sudoer policy to project members just to avoid random confusion
* 03:06 bd808: Added legoktm  to &quot;roots&quot; sudoer policy
* 02:53 bd808: Added legoktm as project admin

=== 2020-01-22 ===
* 11:59 arturo: remove toolviews scripts from toolsbeta-proxy-&lt;nowiki&gt;{&lt;/nowiki&gt;1,2&lt;nowiki&gt;}&lt;/nowiki&gt;, source of cronspam

=== 2020-01-21 ===
* 12:49 arturo: cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1
* 09:40 arturo: livehacking toolsbeta-sgebastion-04 (https://gerrit.wikimedia.org/r/c/566045 and https://gerrit.wikimedia.org/r/c/565575) and toolsbeta-proxy-1 (https://gerrit.wikimedia.org/r/c/565556) for testing [[phab:T234617|T234617]]

=== 2020-01-17 ===
* 12:52 arturo: livehack toolsbeta-puppetmaster-02 to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565556 ([[phab:T234617|T234617]])
* 10:37 arturo: enabling puppet agent in toolsbeta-proxy-1 which was disabled without reason since 2019-12-02 (probably by me)

=== 2020-01-16 ===
* 23:13 bstorm_: updated toollabs-webservice to 0.58 for stretch to test things out
* 12:07 arturo: live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 ([[phab:T242719|T242719]])

=== 2020-01-14 ===
* 02:15 andrewbogott: rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch

=== 2020-01-13 ===
* 16:41 bstorm_: There was a filesystem unclean and other problems on the &quot;old cluster&quot; worker node 1001.  Rebooting it in case that helps.

=== 2020-01-10 ===
* 21:05 bstorm_: updated toollabs-webservice package to 0.55 for testing

=== 2020-01-07 ===
* 15:51 bstorm_: changed kubeadm-config to use a list instead of a hash for extravols on the apiserver in the new k8s cluster [[phab:T242067|T242067]]

=== 2020-01-06 ===
* 21:42 bstorm_: disabled rpcbind on toolsbeta-sgebastion-04 to test some things

=== 2020-01-03 ===
* 17:46 bstorm_: stashed uncommitted changes on the puppetmaster because they seem to be things that are already merged
* 11:27 arturo: [new k8s] cadvisor is running in the metrics namespace now ([[phab:T237643|T237643]])

=== 2020-01-02 ===
* 22:37 bstorm_: Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly.
* 22:19 bstorm_: Changed the ingress-admission ValidatingWebhookConfiguration to check extensions as well as networking API groups

=== 2019-12-17 ===
* 00:14 bstorm_: Fully enabled encryption at rest for toolsbeta kubernetes

=== 2019-12-16 ===
* 23:03 bstorm_: updated the kubeadm-config configmap to match the new init file

=== 2019-12-04 ===
* 13:02 arturo: drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use
* 12:50 arturo: drop puppet prefix `toolsbeta-bastion`, deprecated and no longer in use

=== 2019-12-02 ===
* 10:38 arturo: create wildcard DNS record for `*.toolsbeta.wmflabs.org` for use by the new k8s cluster
* 10:34 arturo: manually scale nginx-ingress deployment to 5 replicas ([[phab:T239405|T239405]])

=== 2019-11-25 ===
* 10:30 arturo: add puppet cert SANs via hiera to toolsbeta-test-k8s-etcd nodes ([[phab:T238655|T238655]])

=== 2019-11-21 ===
* 14:15 arturo: upgrade new k8s cluster to 1.15.6 using kubeadm (plus kubelet)

=== 2019-11-15 ===
* 14:46 arturo: stop live-hacks on toolsbeta-test-k8s-haproxy-1 [[phab:T237643|T237643]]

=== 2019-11-14 ===
* 10:32 arturo: live-hacking toolsbeta-test-k8s-haproxy-1 to point to just the k8s apiserver in control-1 Turn on --v=10 in control-1 for extended debug

=== 2019-11-08 ===
* 19:36 bstorm_: rebooted the proxy server just in case that fixes something.
* 11:58 arturo: adding `profile::toolforge::bastion::nproc: 100` to puppet prefix `toolsbeta-sgebastion` ([[phab:T236202|T236202]])
* 11:38 arturo: new k8s: refresh deployment for nginx-ingress with latest changes from puppet

=== 2019-11-07 ===
* 21:55 bstorm_: killed pods for ingress admission controller to upgrade to new image [[phab:T215531|T215531]]

=== 2019-11-06 ===
* 22:39 bstorm_: upgraded repo version of toollabs-webservice in toolsbeta-stretch to 0.49 -- changes for the new k8s cluster [[phab:T215531|T215531]]
* 19:09 bstorm_: added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis [[phab:T237443|T237443]]
* 18:53 bstorm_: launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone [[phab:T237443|T237443]]
* 18:46 bstorm_: rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica [[phab:T237443|T237443]]
* 18:29 bstorm_: stopped broken kube-proxy service on toolsbeta-proxy-1 (should probably be puppetized)
* 17:35 bstorm_: changing some hiera to work with new proxy host
* 12:44 arturo: created VM toolsbeta-proxy-1 ([[phab:T237443|T237443]])

=== 2019-11-05 ===
* 22:50 bstorm_: deployed the new maintain-kubeusers to toolsbeta [[phab:T215531|T215531]] [[phab:T228499|T228499]]

=== 2019-10-25 ===
* 23:41 bstorm_: Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster [[phab:T215531|T215531]] [[phab:T215678|T215678]] [[phab:T234231|T234231]]
* 16:15 bstorm_: rebooting toolsbeta-test-k8s-worker-1 and -2

=== 2019-10-23 ===
* 12:04 arturo: created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` [[phab:T236074|T236074]]
* 11:56 arturo: point FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` to `toolsbeta-test-k8s-haproxy-1` ([[phab:T236074|T236074]])
* 11:20 arturo: re-create VM `toolsbeta-test-k8s-haproxy-1` to use new puppet profile ([[phab:T236074|T236074]])
* 11:10 arturo: re-create VM `toolsbeta-test-k8s-haproxy-2` to test https://gerrit.wikimedia.org/r/545532 ([[phab:T236074|T236074]])

=== 2019-10-22 ===
* 17:43 arturo: re-create VM `toolsbeta-test-k8s-control-1` [[phab:T236074|T236074]]
* 15:48 arturo: point DNS record `k8s.toolsbeta.eqiad1.wikimedia.cloud` to the first controller node for the bootstrap [[phab:T236074|T236074]]
* 15:30 arturo: created puppet prefix `toolsbeta-test-k8s-control` and delete `toolsbeta-test-k8s-master` [[phab:T236074|T236074]]
* 12:27 arturo: refreshed puppet prefix `toolsbeta-test-k8s-control` with latest info [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=12:26 arturo: created 3 VMs `toolsbeta-test-k8s-control-{1,2,3}` T236074}}
* 12:15 arturo: refresh IP addr of FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` [[phab:T236074|T236074]]
* 12:14 arturo: delete FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=11:57 arturo: created 2 new VMS `toolsbeta-test-k8s-haproxy-{1,2}` T236074}}
* 11:54 arturo: created puppet prefix `toolsbeta-test-k8s-haproxy` and delete `toolsbeta-test-k8s-lb` [[phab:T236074|T236074]]

=== 2019-10-21 ===
* 15:13 arturo: refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers [[phab:T236074|T236074]]
* {{safesubst:SAL entry|1=15:07 arturo: create 3 VMs toolsbeta-test-k8s-etcd-{1,2,3} T236074}}
* 14:58 arturo: deleting all toolsbeta-test-* VMs (master, worker, etcd, lb) [[phab:T236074|T236074]]

=== 2019-10-18 ===
* 16:33 arturo: created DNS zone `toolsbeta.eqiad1.wikimedia.cloud`
* 09:06 arturo: remove puppet prefix toolsbeta-valhallasw-puppet-compiler (unused)
* {{safesubst:SAL entry|1=09:00 arturo: remove puppet prefix toolsbeta-arturo-k8s-{etcd,master,worker} (unused)}}
* {{safesubst:SAL entry|1=08:59 arturo: refresh role for servers in toolsbeta-test-k8s-{master,worker}}}
* 08:58 arturo: remove puppet prefix etcd-k8s-ctest (unused)

=== 2019-10-14 ===
* 12:26 arturo: delete VM `toolsbeta-test-proxy-01` no longer required
* 12:26 arturo: created security group arturo-test-dynamicproxy-backend to tests stuff related to [[phab:T234037|T234037]]

=== 2019-10-09 ===
* 11:59 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster ([[phab:T235059|T235059]])

=== 2019-10-08 ===
* 14:14 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to [[phab:T234037|T234037]]
* 12:27 arturo: created VM toolsbeta-test-proxy-01 for testing stuff related to [[phab:T234037|T234037]]

=== 2019-10-07 ===
* 19:12 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue

=== 2019-09-25 ===
* 23:31 bd808: Updated user list for &quot;roots&quot; sudoer policy
* 23:30 bd808: Granted Krenair projectadmin

=== 2019-09-05 ===
* {{safesubst:SAL entry|1=15:08 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam}}

=== 2019-08-12 ===
* 20:40 phamhi: toolsbeta-test-puppet-sandbox instance created for [[phab:T230147|T230147]]

=== 2019-08-09 ===
* 10:51 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003

=== 2019-07-24 ===
* 20:48 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container [[phab:T228887|T228887]] [[phab:T215531|T215531]]
* 19:02 bstorm_: doing a clean rebuild of the toolsbeta-test-k8s cluster

=== 2019-07-18 ===
* 16:04 arturo: re-create VMs toolsbeta-test-k8s-{master,worker}-*
* 12:47 arturo: create toolsbeta-test-k8s-etcd-2 as buster to check status of latest puppet code ([[phab:T226098|T226098]])
* 12:00 arturo: create toolsbeta-test-k8s-worker-2 as buster to check status of latest puppet code
* {{safesubst:SAL entry|1=09:28 arturo: re-create toolsbeta-test-k8s-master-{1,2,3} as buster to test T228267}}

=== 2019-07-17 ===
* 09:51 arturo: re-create VM toolsbeta-test-k8s-worker-1 as Debian Buster [[phab:T215531|T215531]]
* 09:13 arturo: create VM toolsbeta-test-k8s-master-4 (Debian Buster) [[phab:T215531|T215531]]

=== 2019-07-15 ===
* 12:29 arturo: create `toolsbeta-test-k8s-etcd` puppet prefix
* 12:27 arturo: create `toolsbeta-test-k8s-etcd-1` VM [[phab:T215531|T215531]]

=== 2019-07-03 ===
* 10:49 arturo: recreate `toolsbeta-test-k8s-master-1` VM ([[phab:T215531|T215531]])
* 09:32 arturo: create `toolsbeta-test-k8s-worker-1` VM and a puppet prefix for it ([[phab:T215531|T215531]])
* 09:22 arturo: delete all `toolsbeta-arturo-k8s-*` instances. We no longer require them per new approach at [[phab:T215531|T215531]]

=== 2019-07-02 ===
* 17:24 arturo: `aborrero@toolsbeta-test-k8s-lb-01:~ $ sudo generate_haproxy_default.sh` ([[phab:T215531|T215531]])
* 10:32 arturo: re-creating toolsbeta-test-k8s-master-1 ([[phab:T215531|T215531]]) for it to be created without swap

=== 2019-07-01 ===
* 17:13 arturo: re-creating instance `toolsbeta-test-k8s-master-1` with more CPU for [[phab:T215531|T215531]]
* 17:03 arturo: updated FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` with 172.16.6.9 (the new LB VM) for [[phab:T215531|T215531]]
* 17:02 arturo: re-creating instance `toolsbeta-test-k8s-lb-01` with more CPU for [[phab:T215531|T215531]]
* 16:58 arturo: add puppet prefix `toolsbeta-test-k8s-lb` for [[phab:T215531|T215531]]
* 11:50 arturo: add sssd hiera config for `toolsbeta-test-k8s-master` prefix

=== 2019-06-28 ===
* 19:10 bstorm_: [[phab:T215531|T215531]] removed toolsbeta-arturo-k8s-master-2/3 and added toolsbeta-test-k8s-master-1 for testing kubeadm

=== 2019-06-25 ===
* 10:35 arturo: create puppet prefix `toolsbeta-arturo-k8s-worker` for [[phab:T215531|T215531]]
* 10:35 arturo: create 2 VMs toolsbeta-arturo-k8s-worker-[1,2] for [[phab:T215531|T215531]]

=== 2019-06-21 ===
* 11:42 arturo: re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in [[phab:T226098|T226098]]

=== 2019-06-19 ===
* 10:39 arturo: add myself to the `toolsbeta.admin` LDAP group ([[phab:T225303|T225303]])

=== 2019-06-14 ===
* 16:24 bstorm_: Manually failed &quot;back&quot; to the toolsbeta-sgegrid-master to get the grid functioning again in toolsbeta
* 16:03 bstorm_: [[phab:T221721|T221721]] hard rebooted toolsbeta-sgegrid-master because it had oomkilled basically everything
* 15:55 bstorm_: [[phab:T221721|T221721]] deleted toolsbeta-proxy-01 until it can be actively worked on.
* 15:51 bstorm_: deleted toolsbeta-k8s-lb-01 since it isn't being actively worked on just now

=== 2019-06-06 ===
* 12:14 arturo: [[phab:T215531|T215531]] create 3 VMs `toolsbeta-arturo-k8s-etcd-[1-3]`
* 12:13 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-etcd`* puppet prefix
* 12:12 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-test` puppet prefix

=== 2019-06-05 ===
* 12:40 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets)
* 12:33 arturo: drop VM instances toolsbeta-k8s-master-arturo-[1-3] and create toolsbeta-arturo-k8s-master-[1-3] [[phab:T215531|T215531]]
* 12:32 arturo: drop puppet prefix `toolsbeta-k8s-master-arturo` and create `toolsbeta-arturo-k8s-master` since there is also `toolsbeta-k8s-master` which get applied to my VMs [[phab:T215531|T215531]]
* 11:42 arturo: create VM `toolsbeta-k8s-master-arturo-3` for [[phab:T215531|T215531]] (so I have 3 master nodes in this k8s deployment)
* 11:38 arturo: delete instances arturo-sgeexec-sssd-test-2, arturo-sgeexec-sssd-test-1, arturo-bastion-sssd-test, unused

=== 2019-05-24 ===
* 11:49 arturo: [[phab:T224273|T224273]] create `toolsbeta-k8s-master-arturo` puppet prefix in horizon
* 11:45 arturo: [[phab:T224273|T224273]] create toolsbeta-k8s-master-arturo-[12] stretch VMs
* 11:17 arturo: install by hand some openstack client packages that puppet would refuse to install in toolsbeta-k8s-master-01
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc in toolsbeta-k8s-master-01:
* 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc

=== 2019-05-07 ===
* 10:22 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-exec` puppet prefix
* 10:20 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-generic` puppet prefix
* 10:19 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-lighttpd` puppet prefix

=== 2019-04-25 ===
* 04:17 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver:  toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master

=== 2019-04-12 ===
* 23:34 mutante: - toolsbeta-k8s-master-01 - was out of disk space on / , puppet failed to run because out of disk, rename existing syslog.1.gz, gzip syslog.1, rename existing daemon.log.1.gz, gzip daemong.log.1
* 00:05 andrewbogott: migrating remaining VMs to eqiad1-r

=== 2019-03-25 ===
* 18:00 bd808: All Trusty instances shutdown and now in process of deleting
* 17:42 bd808: Preparing to shutdown beta Trusty job grid

=== 2019-03-22 ===
* 13:59 arturo: create VMs arturo-sgeexec-sssd-test-[12] for testing [[phab:T218126|T218126]]

=== 2019-03-15 ===
* 10:23 arturo: create VM `arturo-bastion-sssd-test` ([[phab:T218126|T218126]])

=== 2019-02-20 ===
* 14:58 andrewbogott: moving toolsbeta-grid-master  and toolsbeta-puppetmaster-02  to labvirt1003

=== 2019-02-14 ===
* 18:30 andrewbogott: moving toolsbeta-puppetdb-01 to labvirt1002

=== 2018-12-04 ===
* 18:43 arturo: some hiera keys reallocated, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477607/

=== 2018-11-26 ===
* 13:26 arturo: [[phab:T210098|T210098]] VM=toolsbeta-sgebastion-03
* 13:25 arturo: [[phab:T210098|T210098]] install systemd239 from stretch-backports and restart VM

=== 2018-11-08 ===
* 10:01 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically [[phab:T207970|T207970]])

=== 2018-10-22 ===
* 21:20 bstorm_: launched a stretch/sonofgridengine master server

=== 2018-09-19 ===
* 20:11 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- [[phab:T200557|T200557]]
* 17:24 bstorm_: new puppetmaster is toolsbeta-puppetmaster-02, however, manual changes are required on each client, so it will be broken for a bit (enabling puppetdb for [[phab:T200557|T200557]])
* 17:06 bstorm_: working on replacing puppetmaster with one running stretch, as part of adding puppetdb

=== 2018-07-22 ===
* 14:28 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug [[phab:T156626|T156626]]

=== 2018-07-18 ===
* 10:46 harej: Deleted toolsbeta-flynn-01

=== 2018-07-12 ===
* 23:06 bstorm_: Got the grid master running

=== 2018-06-28 ===
* 16:34 chicocvenancio: adding harej as root for flynn testing

=== 2018-06-27 ===
* 22:35 chicocvenancio: add harej as project admin to test Flynn stuff

=== 2018-06-22 ===
* 22:26 chicocvenancio: reconfigured toolsbeta-paws-master-01 kubelet to test image pruning
* 09:39 zhuyifei1999_: fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions
* 09:33 zhuyifei1999_: puppet is broken on toolsbeta-bastion-01, investigating
* 09:03 zhuyifei1999_: killing and rebuilding toolsbeta-bastion-01
* 08:31 zhuyifei1999_: on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice
* 07:56 andrewbogott: someone removed /usr/bin/webservice

=== 2018-05-15 ===
* 07:26 zhuyifei1999_: applied {{Gerrit|5324236}} via toolsbeta-puppetmaster-01 [[phab:T190893|T190893]]
* 05:28 zhuyifei1999_: Making project puppetmaster at toolsbeta-puppetmaster-01

=== 2018-05-08 ===
* 02:18 zhuyifei1999_: manually created flannel etcd key [[phab:T190893|T190893]]

=== 2018-05-07 ===
* 19:01 zhuyifei1999_: install kubernetes-client on toolsbeta-worker-1001 to debug stuffs
* 18:41 zhuyifei1999_: rebuilding toolsbeta-k8s-etcd-01
* 17:58 zhuyifei1999_: cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 {{!}} (while read dir; do id toolsbeta.`basename $dir` 2&gt; /dev/null {{!}}{{!}} sudo rm -rfv $dir; done)`
* 16:41 zhuyifei1999_: rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd

=== 2018-05-06 ===
* 04:06 zhuyifei1999_: locally patched `/usr/lib/python2.7/dist-packages/toollabs/common/tool.py` on bastion and webgrid-lighttpd

=== 2018-05-05 ===
* 19:51 zhuyifei1999_: `systemctl mask maintain-kubeusers` because it's causing a mess, tries to get the tool list from toolforge [[phab:T190893|T190893]]
* 18:40 zhuyifei1999_: to unblock k8s testing while waiting on https://gerrit.wikimedia.org/r/430539, installed the package directly on `toolsbeta-k8s-master-01` with `$ sudo apt install python3-yaml`

=== 2018-05-02 ===
* 21:02 zhuyifei1999_: copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera
* 20:37 bd808: Added Neha16 as a project admin for work on [[phab:T175768|T175768]]
* 20:31 zhuyifei1999_: nuke webservice instances and rebuild
* 20:31 zhuyifei1999_: Added k8s_infrastructure_users to project hiera on horizon [[phab:T192618|T192618]]

=== 2018-04-20 ===
* 00:20 zhuyifei1999_: deleted all instances I just created except k8s master because of chicken-and-egg problem

=== 2018-04-19 ===
* 22:10 zhuyifei1999_: the trusty instances ask me for my password. the jessie instances don't like my ssh key. :(
* 21:59 zhuyifei1999_: got 'Error: RecordSet belongs in a child zone: toolsbeta.wmflabs.org', using tools-beta.wmflabs.org instead
* 21:57 zhuyifei1999_: Add proxy toolsbeta.wmflabs.org =&gt; toolsbeta-proxy-01.toolsbeta.eqiad.wmflabs
* 21:43 zhuyifei1999_: Start creating instances for webservice setup [[phab:T190893|T190893]]

=== 2018-03-30 ===
* 22:40 zhuyifei1999_: copied over many prefix puppet configuration in horizon from toolforge [[phab:T190893|T190893]]

=== 2018-03-14 ===
* 18:07 chicocvenancio: updated paws-beta k8s cluster  and nodes  to v1.9.4 for [[phab:T189680|T189680]]

=== 2018-03-05 ===
* 19:33 chicocvenancio: added Zhuyifei1999 as project admin

=== 2018-02-09 ===
* 01:11 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])

=== 2017-08-07 ===
* 14:09 andrewbogott: deleted etcd-k8s-CTEST  and k8s-master-CTEST

=== 2017-04-26 ===
* 15:38 madhuvishy: add Madhuvishy as projectadmin

=== 2016-10-07 ===
* 19:30 valhallasw`cloud: (puppet certs, to be precise)
* 19:30 valhallasw`cloud: fixed certs on toolsbeta-vagrant3-scfc.toolsbeta.eqiad.wmflabs

=== 2016-10-04 ===
* 19:31 valhallasw`cloud: puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for)

=== 2016-09-08 ===
* 17:11 bd808: Added BryanDavis (self) to project as admin

=== 2016-08-29 ===
* 19:20 yuvipanda: reboot toolsbeta-master, seems, uh, stuck
* 19:18 yuvipanda: reboot toolsbeta-mail, seems, uh, stuck
* 18:48 yuvipanda: reboot toolsbeta-puppetmaster3, puppet run process became Zommmmbiiiieeee, ate all my brains

=== 2016-07-03 ===
* 15:02 yuvipanda: migrating toolsbeta-valhallasw-puppet-compiler to labvirt1011 to ease pressure on labvirt1010

=== 2016-05-27 ===
* 18:57 valhallasw`cloud: sudo qconf -Ae /var/lib/gridengine/etc/exechosts/toolsbeta-exec-1209.toolsbeta.eqiad.wmflabs

=== 2016-05-26 ===
* 15:08 valhallasw`cloud: toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host

=== 2015-10-13 ===
* 19:21 valhallasw`cloud: started building toolsbeta-bastion.

=== 2015-09-07 ===
* 18:50 valhallasw`cloud: role::bastion is now applied on -exec-101. Now for the package_builder manifest...
* 18:30 valhallasw`cloud: applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages)

=== July 4 ===
* 12:57 valhallasw`cloud: restarting toolsbeta-webproxy, no response on port 22

=== July 2 ===
* 14:55 valhallasw`cloud: toolsbeta-webproxy does not respond at all to SSH; rebooting

=== July 1 ===
* 19:47 valhallasw`cloud: still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it?
* 19:44 valhallasw`cloud: restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login

=== June 7 ===
* 14:44 valhallasw: updated /var/lib/git/operations/puppet to make sure the other hosts get the memo
* 14:42 YuviPanda: run sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on toolsbeta-puppetmaster3 to fix broken LDAP TLS config

=== May 11 ===
* 18:14 valhallasw: building toolsbeta-pbuilder to experiment with pbuilder for building packages

=== May 2 ===
* 11:11 valhallasw`cloud: commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code
* 10:37 valhallasw`cloud: that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera
* 10:30 valhallasw`cloud: pulled new changes into puppetmaster to get https://github.com/wikimedia/operations-puppet/commit/4afd23d8e2905a84ef211ad92e8314173eb743ba in
* 10:25 valhallasw`cloud: set Hiera variable &quot;elasticsearch::cluster_name&quot;: toolsbeta-logstash-eqiad
* 10:09 valhallasw`cloud: created [[Nova_Resource:I-00000c01.eqiad.wmflabs|toolsbeta-logstash]] to play around with logstash and figure out what we need for tools ([[phab:T97861]])

=== April 26 ===
* 18:18 valhallasw`cloud: having some issues with puppet-test, so postponing for now
* 17:12 valhallasw`cloud: deploying https://gerrit.wikimedia.org/r/#/c/206118/ on tools-beta using puppet-test

=== March 31 ===
* 00:27 andrewbogott: shut down toolsbeta-webgrid-03 to conserve resources.  It can be restarted when needed.

=== September 20 ===
* 20:09 andrewbogott_afk: moved toolsbeta-exec-01 and toolsbeta-scfc-icinga-test off of virt1006

=== July 22 ===
* 11:36 scfc_de: Removed andrewbogott_afk, Coren, petan, YuviPanda from service group admin to prevent further spamming :-)

=== August 19 ===
* 12:44 petan: rebooting apache it seems to be frozen

=== August 4 ===
* 23:50 scfc_de: Added scfc_de to local-admin so I don't log myself out again :-)

=== July 6 ===
* 19:42 petan: rebooting login

=== June 26 ===
* 08:03 wm-bot: petrb: updating logsplitter

=== June 24 ===
* 14:47 wm-bot: petrb: rebooting exec-01 to fix the grid weird info
* 13:43 scfc_de: Made scfc root.
* 13:42 scfc_de: Created toolsbeta-puppetmaster.
* 11:09 YuviPanda: Granted yuvipanda root on toolsbeta

=== June 21 ===
* 13:46 wm-bot: petrb: rebooting all servers

=== June 17 ===
* 08:31 petan: switching all instances to nfs

=== June 16 ===
* 15:37 petan: importing sudo policies of tools
* 15:36 petan: importing security groups of tools
* 15:36 petan: blah

{{SAL|Project Name=toolsbeta}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>hb4nb086xp3h99x7r1pl7zv973suxny</sha1>
    </revision>
  </page>
  <page>
    <title>Server Admin Log</title>
    <ns>0</ns>
    <id>7919</id>
    <revision>
      <id>2309619</id>
      <parentid>2309609</parentid>
      <timestamp>2025-06-08T09:53:23Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.data-reload (exit_code=0) reloading scholarly_articles on wdqs1023.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/scholarly/20250526/ using stat1011.eqiad.wmnet)</comment>
      <origin>2309619</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="286625" sha1="hr8tush4ari9615doj18zd6yua28rfs" xml:space="preserve">== 2025-06-08 ==
* 09:53 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.data-reload (exit_code=0) reloading scholarly_articles on wdqs1023.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/scholarly/20250526/ using stat1011.eqiad.wmnet)

== 2025-06-07 ==
* 11:43 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be2007.codfw.wmnet with OS bullseye
* 11:07 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be2006.codfw.wmnet with OS bullseye
* 08:12 elukey: restart apache2 / php-fpm on phab1004
* 04:18 mutante: restarted apache on phab1004

== 2025-06-06 ==
* 21:33 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:25 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:19 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:15 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:02 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on relforge[1003-1004].eqiad.wmnet with reason: downtime before decom
* 20:42 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be2007.codfw.wmnet with reason: host reimage
* 20:40 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 20:38 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be2007.codfw.wmnet with reason: host reimage
* 20:35 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 20:16 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 20:15 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host thanos-be2007.codfw.wmnet with OS bullseye
* 20:06 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be2006.codfw.wmnet with reason: host reimage
* 20:03 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be2006.codfw.wmnet with reason: host reimage
* 19:45 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1185.eqiad.wmnet with OS bullseye
* 19:31 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host thanos-be2006.codfw.wmnet with OS bullseye
* 19:11 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T383811|T383811]] - bking@cumin2002
* 19:01 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2244.codfw.wmnet with OS bookworm
* 19:01 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 18:25 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1185.eqiad.wmnet with OS bullseye
* 18:06 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002&quot;
* 17:49 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2244.codfw.wmnet with reason: host reimage
* 17:46 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2244.codfw.wmnet with reason: host reimage
* 17:29 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host db2244.codfw.wmnet with OS bookworm
* 17:29 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2244']
* 17:29 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2244']
* 17:20 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T383811|T383811]] - bking@cumin2002
* 17:10 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2244.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 17:08 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T383811|T383811]] - bking@cumin2002
* 17:06 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: [[phab:T383811|T383811]] - bking@cumin2002
* 17:00 sukhe: forced agent run on O:alerting_host to reload vopsbot to add cdobbins
* 16:57 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2244.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 16:57 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2244.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 16:56 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2244.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 16:55 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2244
* 16:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host db2244
* 16:44 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:42 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 16:08 sbassett: Deployed security update to fix [[phab:T396111|T396111]]
* 15:41 eevans@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2066.codfw.wmnet
* 15:34 eevans@cumin1002: START - Cookbook sre.hosts.reboot-single for host ms-be2066.codfw.wmnet
* 15:24 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:23 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.admin config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 15:19 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1013.eqiad.wmnet
* 15:19 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs1013.eqiad.wmnet
* 14:53 akosiaris@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:42 akosiaris@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:24 sukhe@dns1004: END - running authdns-update
* 14:23 sukhe@dns1004: START - running authdns-update
* 14:23 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns2004*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 14:23 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot finished rebooting dns2004.wikimedia.org
* 14:22 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns1005*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 14:22 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot finished rebooting dns1005.wikimedia.org
* 14:10 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot begin reboot of dns1005.wikimedia.org
* 14:10 sukhe@cumin1002: START - Cookbook sre.dns.roll-reboot rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns1005*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 14:10 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot begin reboot of dns2004.wikimedia.org
* 14:10 sukhe@cumin1002: START - Cookbook sre.dns.roll-reboot rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns2004*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 13:51 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns2006*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 13:51 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot finished rebooting dns2006.wikimedia.org
* 13:49 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns1006*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 13:49 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot finished rebooting dns1006.wikimedia.org
* 13:40 vgutierrez@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 13:40 vgutierrez@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 13:40 vgutierrez@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs1013.eqiad.wmnet with reason: switching to katran
* 13:35 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot begin reboot of dns2006.wikimedia.org
* 13:35 sukhe@cumin1002: START - Cookbook sre.dns.roll-reboot rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns2006*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 13:34 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot begin reboot of dns1006.wikimedia.org
* 13:34 sukhe@cumin1002: START - Cookbook sre.dns.roll-reboot rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns1006*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 13:32 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns3003*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 13:32 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot finished rebooting dns3003.wikimedia.org
* 13:31 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns6001*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 13:31 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot finished rebooting dns6001.wikimedia.org
* 13:21 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:21 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add back entry for mistakenly deleted ssw1-a8-codfw IP - cmooney@cumin1003&quot;
* 13:21 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add back entry for mistakenly deleted ssw1-a8-codfw IP - cmooney@cumin1003&quot;
* 13:18 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot begin reboot of dns6001.wikimedia.org
* 13:18 sukhe@cumin1002: START - Cookbook sre.dns.roll-reboot rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns6001*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 13:18 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot begin reboot of dns3003.wikimedia.org
* 13:18 sukhe@cumin1002: START - Cookbook sre.dns.roll-reboot rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns3003*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 13:13 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns5003*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 13:13 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot finished rebooting dns5003.wikimedia.org
* 13:09 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns4003*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 13:09 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot finished rebooting dns4003.wikimedia.org
* 13:06 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 12:58 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot begin reboot of dns5003.wikimedia.org
* 12:58 sukhe@cumin1002: START - Cookbook sre.dns.roll-reboot rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns5003*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 12:58 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot begin reboot of dns4003.wikimedia.org
* 12:58 sukhe@cumin1002: START - Cookbook sre.dns.roll-reboot rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;dns4003*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:dnsbox)
* 12:20 fceratto@cumin1002: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) db2158 - Depool db2158.codfw.wmnet to then clone it to db2151.codfw.wmnet - fceratto@cumin1002
* 12:20 fceratto@cumin1002: START - Cookbook sre.mysql.depool db2158 - Depool db2158.codfw.wmnet to then clone it to db2151.codfw.wmnet - fceratto@cumin1002
* 12:19 fceratto@cumin1002: START - Cookbook sre.mysql.clone of db2158.codfw.wmnet onto db2151.codfw.wmnet
* 12:03 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 12:03 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 11:52 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2151.codfw.wmnet with reason: Disabling notifications
* 11:42 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 11:42 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 11:32 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 11:32 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 11:32 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 11:31 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 11:31 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 11:30 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 11:30 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 11:28 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 11:28 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:28 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:28 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:27 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:27 jayme@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:26 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:25 jayme@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:22 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 11:21 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 11:18 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 11:18 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 11:17 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 11:17 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 11:12 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 11:12 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 09:40 fceratto@cumin1002: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) db2151* - Log issue and disk filled up
* 09:40 fceratto@cumin1002: START - Cookbook sre.mysql.depool db2151* - Log issue and disk filled up
* 09:05 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 09:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:37 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 398044
* 08:36 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 398044
* 08:35 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 46997
* 08:34 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 46997
* 08:33 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 394065
* 08:33 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 394065
* 08:32 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 46562
* 08:32 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 46562
* 08:30 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13150
* 08:30 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 13150
* 08:28 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 199524
* 08:25 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 199524
* 08:24 volans@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts cirrussearch2113.codfw.wmnet
* 08:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 63199
* 08:23 volans@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cirrussearch2113.codfw.wmnet
* 08:23 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 63199
* 08:22 volans@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts cirrussearch2113.codfw.wmnet
* 08:22 volans@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cirrussearch2113.codfw.wmnet
* 08:11 volans@cumin1003: START - Cookbook sre.hosts.reboot-single for host cirrussearch2113.codfw.wmnet
* 08:10 volans@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cirrussearch2113.codfw.wmnet
* 07:52 ryankemper@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on cirrussearch2113.codfw.wmnet with reason: [[phab:T394543|T394543]]
* 06:15 stevemunene@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker[1157-1159].eqiad.wmnet with reason: Upgrade an-worker hard drives from 4TB to 8TB group 5 - rack F1
* 06:14 stevemunene@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker[1160-1162].eqiad.wmnet with reason: Upgrade an-worker hard drives from 4TB to 8TB group 5 - rack F1
* 06:13 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-b3-magru
* 06:13 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device asw1-b3-magru
* 05:42 XioNoX: push pfw policies - [[phab:T395904|T395904]]
* 03:02 eevans@cumin1002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw
* 02:57 eevans@cumin1002: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw

== 2025-06-05 ==
* 22:05 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 22:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 22:03 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 22:02 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:57 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:42 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:24 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:00 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:00 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 21:00 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 20:59 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 20:57 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 20:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 20:56 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 20:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 20:50 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 20:49 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 20:24 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1154098{{!}}Fix back compat for data-chart (T395462)]] (duration: 10m 05s)
* 20:17 jdlrobson@deploy1003: jdlrobson: Continuing with sync
* 20:16 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1154098{{!}}Fix back compat for data-chart (T395462)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:14 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1154098{{!}}Fix back compat for data-chart (T395462)]]
* 20:09 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1185.eqiad.wmnet with OS bullseye
* 19:24 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:15 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:14 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:14 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1186
* 19:13 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1186
* 19:13 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:12 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1186
* 19:12 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1186
* 18:52 phuedx: Disabled the SDS 2.4.11 Synthetic A/A Test in xLab
* 18:49 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1185.eqiad.wmnet with OS bullseye
* 18:48 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:43 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:32 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-worker1185.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:30 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:21 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.4  refs [[phab:T392174|T392174]]
* 18:21 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:20 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1186
* 18:20 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1186
* 18:19 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1185.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:18 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 18:17 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 18:17 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-reload reloading scholarly_articles on wdqs1023.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/scholarly/20250526/ using stat1011.eqiad.wmnet)
* 18:17 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-worker1185.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:17 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1185.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 18:05 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 18:04 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 17:28 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 17:27 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 17:25 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 17:24 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 17:24 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 17:24 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 17:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 17:21 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.admin config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 17:15 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1013.eqiad.wmnet
* 17:15 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs1013.eqiad.wmnet
* 16:54 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for acmechief2002.codfw.wmnet,acmechief1002.eqiad.wmnet,acmechief-test2001.codfw.wmnet,acmechief-test1001.eqiad.wmnet
* 16:54 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for acmechief2002.codfw.wmnet,acmechief1002.eqiad.wmnet,acmechief-test2001.codfw.wmnet,acmechief-test1001.eqiad.wmnet
* 16:51 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@930d28b]: adapt check_bad_parsing to dumps 2.0 (duration: 01m 16s)
* 16:50 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@930d28b]: adapt check_bad_parsing to dumps 2.0
* 16:50 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on acmechief2002.codfw.wmnet,acmechief1002.eqiad.wmnet,acmechief-test2001.codfw.wmnet,acmechief-test1001.eqiad.wmnet with reason: Reboots
* 16:27 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153750{{!}}Revert &quot;Deploy survey to en at twenty percent&quot;]] (duration: 11m 23s)
* 16:20 jdlrobson@deploy1003: jdlrobson, jdrewniak: Continuing with sync
* 16:18 jdlrobson@deploy1003: jdlrobson, jdrewniak: Backport for [[gerrit:1153750{{!}}Revert &quot;Deploy survey to en at twenty percent&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:17 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77191 and previous config saved to /var/cache/conftool/dbconfig/20250605-161701-fceratto.json
* 16:16 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1153750{{!}}Revert &quot;Deploy survey to en at twenty percent&quot;]]
* 16:12 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2244.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 16:03 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2244.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 16:02 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2244
* 16:02 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host db2244
* 16:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P77190 and previous config saved to /var/cache/conftool/dbconfig/20250605-160154-fceratto.json
* 16:01 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:01 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: adding db2244 to codfw - jhancock@cumin2002&quot;
* 16:01 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: adding db2244 to codfw - jhancock@cumin2002&quot;
* 15:57 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:55 aokoth@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on doc1003.eqiad.wmnet with reason: Bookworm Migration
* 15:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P77189 and previous config saved to /var/cache/conftool/dbconfig/20250605-154647-fceratto.json
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2237 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77188 and previous config saved to /var/cache/conftool/dbconfig/20250605-153139-fceratto.json
* 15:23 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2237 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77187 and previous config saved to /var/cache/conftool/dbconfig/20250605-152314-fceratto.json
* 15:23 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance
* 15:22 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77186 and previous config saved to /var/cache/conftool/dbconfig/20250605-152248-fceratto.json
* 15:08 taavi@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2004-dev.codfw.wmnet
* 15:07 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P77185 and previous config saved to /var/cache/conftool/dbconfig/20250605-150741-fceratto.json
* 14:59 taavi@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudlb2004-dev.codfw.wmnet
* 14:59 taavi@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2003-dev.codfw.wmnet
* 14:53 damilare: payments-wiki upgraded from {{Gerrit|2d8b655a}} to {{Gerrit|aa102260}}
* 14:52 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P77184 and previous config saved to /var/cache/conftool/dbconfig/20250605-145234-fceratto.json
* 14:50 taavi@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudlb2003-dev.codfw.wmnet
* 14:37 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2236 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77183 and previous config saved to /var/cache/conftool/dbconfig/20250605-143724-fceratto.json
* 14:29 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2236 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77182 and previous config saved to /var/cache/conftool/dbconfig/20250605-142908-fceratto.json
* 14:29 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance
* 14:28 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77181 and previous config saved to /var/cache/conftool/dbconfig/20250605-142840-fceratto.json
* 14:20 taavi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) 'private.codfw.wikimedia.cloud$' on codfw recursors
* 14:20 taavi@cumin1002: START - Cookbook sre.dns.wipe-cache 'private.codfw.wikimedia.cloud$' on codfw recursors
* 14:19 tgr@deploy1003: Unlocked for deployment [MediaWiki]: [[phab:T395468|T395468]] (duration: 39m 39s)
* 14:19 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs1013.eqiad.wmnet with reason: switching to katran
* 14:18 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 14:18 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.admin config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 14:18 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host sretest2007
* 14:18 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host sretest2007
* 14:17 taavi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:17 taavi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: assign cloud-private v6 addresses for codfw1dev devices - taavi@cumin1002&quot;
* 14:17 taavi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: assign cloud-private v6 addresses for codfw1dev devices - taavi@cumin1002&quot;
* 14:17 tgr: deploying a PrivateSettings config change
* 14:13 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P77179 and previous config saved to /var/cache/conftool/dbconfig/20250605-141333-fceratto.json
* 14:13 taavi@cumin1002: START - Cookbook sre.dns.netbox
* 14:11 marostegui@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77178 and previous config saved to /var/cache/conftool/dbconfig/20250605-141108-root.json
* 13:58 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P77177 and previous config saved to /var/cache/conftool/dbconfig/20250605-135826-fceratto.json
* 13:56 marostegui@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77176 and previous config saved to /var/cache/conftool/dbconfig/20250605-135603-root.json
* 13:51 marostegui: Migrate s2 codfw to SBR dbmaint [[phab:T383795|T383795]]
* 13:50 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 13:50 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2037.codfw.wmnet
* 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 13:49 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2037.codfw.wmnet
* 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 13:49 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 13:49 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
* 13:48 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
* 13:48 vgutierrez: upload liberica 0.16 to bookworm-wikimedia (apt.wm.o) - [[phab:T395228|T395228]]
* 13:48 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 13:47 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 13:44 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2037.codfw.wmnet
* 13:43 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77175 and previous config saved to /var/cache/conftool/dbconfig/20250605-134319-fceratto.json
* 13:41 marostegui@cumin1002: dbctl commit (dc=all): 'es2042 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77174 and previous config saved to /var/cache/conftool/dbconfig/20250605-134158-root.json
* 13:41 marostegui@cumin1002: dbctl commit (dc=all): 'es2045 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77173 and previous config saved to /var/cache/conftool/dbconfig/20250605-134153-root.json
* 13:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77172 and previous config saved to /var/cache/conftool/dbconfig/20250605-134057-root.json
* 13:40 moritzm: installing net-tools bugfix updates for bookworm
* 13:40 tgr@deploy1003: Locking from deployment [MediaWiki]: [[phab:T395468|T395468]]
* 13:38 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2037.codfw.wmnet
* 13:37 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2036.codfw.wmnet
* 13:37 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2036.codfw.wmnet
* 13:35 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2219 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77171 and previous config saved to /var/cache/conftool/dbconfig/20250605-133500-fceratto.json
* 13:34 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance
* 13:34 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77170 and previous config saved to /var/cache/conftool/dbconfig/20250605-133434-fceratto.json
* 13:31 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2036.codfw.wmnet
* 13:26 marostegui@cumin1002: dbctl commit (dc=all): 'es2042 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77169 and previous config saved to /var/cache/conftool/dbconfig/20250605-132652-root.json
* 13:26 marostegui@cumin1002: dbctl commit (dc=all): 'es2045 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77168 and previous config saved to /var/cache/conftool/dbconfig/20250605-132648-root.json
* 13:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77167 and previous config saved to /var/cache/conftool/dbconfig/20250605-132552-root.json
* 13:23 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netflow7001.magru.wmnet
* 13:23 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:23 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: netflow7001.magru.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 13:21 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: netflow7001.magru.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 13:21 Lucas_WMDE: UTC afternoon backport+config window done
* 13:19 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P77166 and previous config saved to /var/cache/conftool/dbconfig/20250605-131926-fceratto.json
* 13:18 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2036.codfw.wmnet
* 13:17 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 13:16 gkyziridis@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153945{{!}}ores-extension: enable extension with revertrisk filter for second batch of wikis (excluding azwiki) (T395823)]] (duration: 11m 51s)
* 13:12 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts netflow7001.magru.wmnet
* 13:11 marostegui@cumin1002: dbctl commit (dc=all): 'es2042 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77165 and previous config saved to /var/cache/conftool/dbconfig/20250605-131147-root.json
* 13:11 marostegui@cumin1002: dbctl commit (dc=all): 'es2045 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77164 and previous config saved to /var/cache/conftool/dbconfig/20250605-131142-root.json
* 13:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77163 and previous config saved to /var/cache/conftool/dbconfig/20250605-131046-root.json
* 13:09 gkyziridis@deploy1003: gkyziridis: Continuing with sync
* 13:07 gkyziridis@deploy1003: gkyziridis: Backport for [[gerrit:1153945{{!}}ores-extension: enable extension with revertrisk filter for second batch of wikis (excluding azwiki) (T395823)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:04 gkyziridis@deploy1003: Started scap sync-world: Backport for [[gerrit:1153945{{!}}ores-extension: enable extension with revertrisk filter for second batch of wikis (excluding azwiki) (T395823)]]
* 13:04 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P77162 and previous config saved to /var/cache/conftool/dbconfig/20250605-130419-fceratto.json
* 13:03 taavi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;remove outdated octavia net - taavi@cumin1002&quot;
* 13:03 taavi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;remove outdated octavia net - taavi@cumin1002&quot;
* 12:56 marostegui@cumin1002: dbctl commit (dc=all): 'es2042 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77161 and previous config saved to /var/cache/conftool/dbconfig/20250605-125641-root.json
* 12:56 marostegui@cumin1002: dbctl commit (dc=all): 'es2045 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77160 and previous config saved to /var/cache/conftool/dbconfig/20250605-125637-root.json
* 12:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77159 and previous config saved to /var/cache/conftool/dbconfig/20250605-125540-root.json
* 12:54 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2035.codfw.wmnet
* 12:54 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2035.codfw.wmnet
* 12:51 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2151.codfw.wmnet with reason: Maintenance
* 12:50 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2151 [[phab:T395989|T395989]]', diff saved to https://phabricator.wikimedia.org/P77158 and previous config saved to /var/cache/conftool/dbconfig/20250605-125057-marostegui.json
* 12:49 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77157 and previous config saved to /var/cache/conftool/dbconfig/20250605-124912-fceratto.json
* 12:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2035.codfw.wmnet
* 12:43 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2035.codfw.wmnet
* 12:41 marostegui@cumin1002: dbctl commit (dc=all): 'es2042 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77156 and previous config saved to /var/cache/conftool/dbconfig/20250605-124136-root.json
* 12:41 marostegui@cumin1002: dbctl commit (dc=all): 'es2045 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77155 and previous config saved to /var/cache/conftool/dbconfig/20250605-124131-root.json
* 12:41 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2210 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77154 and previous config saved to /var/cache/conftool/dbconfig/20250605-124110-fceratto.json
* 12:41 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance
* 12:40 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77153 and previous config saved to /var/cache/conftool/dbconfig/20250605-124043-fceratto.json
* 12:32 jakob@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 12:32 jakob@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 12:31 jakob@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 12:30 jakob@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 12:30 jakob@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 12:30 jakob@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 12:26 marostegui@cumin1002: dbctl commit (dc=all): 'es2042 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77151 and previous config saved to /var/cache/conftool/dbconfig/20250605-122631-root.json
* 12:26 marostegui@cumin1002: dbctl commit (dc=all): 'es2045 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77150 and previous config saved to /var/cache/conftool/dbconfig/20250605-122625-root.json
* 12:25 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P77149 and previous config saved to /var/cache/conftool/dbconfig/20250605-122537-fceratto.json
* 12:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es[2042,2045].codfw.wmnet with reason: Maintenance
* 12:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2042 es2045 [[phab:T395241|T395241]]', diff saved to https://phabricator.wikimedia.org/P77147 and previous config saved to /var/cache/conftool/dbconfig/20250605-122035-marostegui.json
* 12:10 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P77146 and previous config saved to /var/cache/conftool/dbconfig/20250605-121029-fceratto.json
* 11:55 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77145 and previous config saved to /var/cache/conftool/dbconfig/20250605-115522-fceratto.json
* 11:49 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 11:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 11:47 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2206 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77144 and previous config saved to /var/cache/conftool/dbconfig/20250605-114711-fceratto.json
* 11:47 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance
* 11:42 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2199.codfw.wmnet with reason: Maintenance
* 11:42 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77143 and previous config saved to /var/cache/conftool/dbconfig/20250605-114213-fceratto.json
* 11:35 moritzm: installing Linux 5.10.237 on Bullseye hosts
* 11:27 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P77142 and previous config saved to /var/cache/conftool/dbconfig/20250605-112706-fceratto.json
* 11:26 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2032.codfw.wmnet
* 11:25 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2032.codfw.wmnet
* 11:25 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77141 and previous config saved to /var/cache/conftool/dbconfig/20250605-112518-root.json
* 11:25 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77140 and previous config saved to /var/cache/conftool/dbconfig/20250605-112511-root.json
* 11:19 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2032.codfw.wmnet
* 11:11 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P77139 and previous config saved to /var/cache/conftool/dbconfig/20250605-111158-fceratto.json
* 11:10 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77138 and previous config saved to /var/cache/conftool/dbconfig/20250605-111013-root.json
* 11:10 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77137 and previous config saved to /var/cache/conftool/dbconfig/20250605-111005-root.json
* 11:09 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2032.codfw.wmnet
* 11:04 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet
* 11:03 ryankemper@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading scholarly_articles on wdqs1023.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/scholarly/20250526/ using stat1011.eqiad.wmnet)
* 11:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2031.codfw.wmnet
* 11:02 gehel: restarting Blazegraph on wdqs1023 to address allocator decreasing alert
* 10:57 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2031.codfw.wmnet
* 10:56 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2179 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77136 and previous config saved to /var/cache/conftool/dbconfig/20250605-105650-fceratto.json
* 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77135 and previous config saved to /var/cache/conftool/dbconfig/20250605-105507-root.json
* 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77134 and previous config saved to /var/cache/conftool/dbconfig/20250605-105500-root.json
* 10:52 marostegui@cumin1002: dbctl commit (dc=all): 'db2158 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77133 and previous config saved to /var/cache/conftool/dbconfig/20250605-105216-root.json
* 10:49 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2179 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77132 and previous config saved to /var/cache/conftool/dbconfig/20250605-104928-fceratto.json
* 10:49 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance
* 10:49 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77131 and previous config saved to /var/cache/conftool/dbconfig/20250605-104912-fceratto.json
* 10:42 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet
* 10:41 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2030.codfw.wmnet
* 10:40 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77130 and previous config saved to /var/cache/conftool/dbconfig/20250605-104002-root.json
* 10:39 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77129 and previous config saved to /var/cache/conftool/dbconfig/20250605-103954-root.json
* 10:39 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1165.eqiad.wmnet
* 10:37 marostegui@cumin1002: dbctl commit (dc=all): 'db2158 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77128 and previous config saved to /var/cache/conftool/dbconfig/20250605-103711-root.json
* 10:34 tappof@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host centrallog2002.codfw.wmnet
* 10:34 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P77127 and previous config saved to /var/cache/conftool/dbconfig/20250605-103403-fceratto.json
* 10:33 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet
* 10:32 stevemunene@cumin1002: START - Cookbook sre.hosts.reboot-single for host an-worker1165.eqiad.wmnet
* 10:31 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1164.eqiad.wmnet
* 10:31 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
* 10:30 Ammar: Ran fixStuckGlobalRename.php for [[phab:T396054|T396054]]
* 10:27 claime: Manual run of generatecaptcha on mw-cron with delete - [[phab:T388531|T388531]]
* 10:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 10:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 10:24 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77126 and previous config saved to /var/cache/conftool/dbconfig/20250605-102456-root.json
* 10:24 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77125 and previous config saved to /var/cache/conftool/dbconfig/20250605-102449-root.json
* 10:24 stevemunene@cumin1002: START - Cookbook sre.hosts.reboot-single for host an-worker1164.eqiad.wmnet
* 10:23 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1163.eqiad.wmnet
* 10:23 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es2041 to es4 master and es2044 as es5 master', diff saved to https://phabricator.wikimedia.org/P77124 and previous config saved to /var/cache/conftool/dbconfig/20250605-102319-root.json
* 10:22 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet
* 10:22 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2029.codfw.wmnet
* 10:22 marostegui@cumin1002: dbctl commit (dc=all): 'db2158 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77123 and previous config saved to /var/cache/conftool/dbconfig/20250605-102205-root.json
* 10:21 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
* 10:18 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P77122 and previous config saved to /var/cache/conftool/dbconfig/20250605-101856-fceratto.json
* 10:16 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2029.codfw.wmnet
* 10:16 stevemunene@cumin1002: START - Cookbook sre.hosts.reboot-single for host an-worker1163.eqiad.wmnet
* 10:09 tappof@cumin1002: START - Cookbook sre.hosts.reboot-single for host centrallog2002.codfw.wmnet
* 10:09 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77121 and previous config saved to /var/cache/conftool/dbconfig/20250605-100950-root.json
* 10:09 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77120 and previous config saved to /var/cache/conftool/dbconfig/20250605-100943-root.json
* 10:08 cgoubert@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153937{{!}}Revert &quot;ores-extension: enable extension with revertrisk filter for second batch of wikis&quot;]] (duration: 10m 36s)
* 10:08 tappof@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2007.codfw.wmnet
* 10:07 marostegui@cumin1002: dbctl commit (dc=all): 'db2158 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77119 and previous config saved to /var/cache/conftool/dbconfig/20250605-100700-root.json
* 10:05 marostegui@cumin1002: dbctl commit (dc=all): 'db2169 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77118 and previous config saved to /var/cache/conftool/dbconfig/20250605-100527-root.json
* 10:04 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es[2043,2046].codfw.wmnet with reason: Maintenance
* 10:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2043 es2046 [[phab:T395241|T395241]]', diff saved to https://phabricator.wikimedia.org/P77117 and previous config saved to /var/cache/conftool/dbconfig/20250605-100419-marostegui.json
* 10:04 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@d11bd51]: Update webrequest-test hive jar for ua-parser (duration: 00m 16s)
* 10:03 joal@deploy1003: Started deploy [airflow-dags/analytics_test@d11bd51]: Update webrequest-test hive jar for ua-parser
* 10:03 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77116 and previous config saved to /var/cache/conftool/dbconfig/20250605-100349-fceratto.json
* 10:03 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet
* 10:01 cgoubert@deploy1003: gkyziridis, cgoubert: Continuing with sync
* 10:00 tappof@cumin1002: START - Cookbook sre.hosts.reboot-single for host prometheus2007.codfw.wmnet
* 10:00 cgoubert@deploy1003: gkyziridis, cgoubert: Backport for [[gerrit:1153937{{!}}Revert &quot;ores-extension: enable extension with revertrisk filter for second batch of wikis&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:58 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2028.codfw.wmnet
* 09:58 tappof@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1007.eqiad.wmnet
* 09:58 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2028.codfw.wmnet
* 09:57 cgoubert@deploy1003: Started scap sync-world: Backport for [[gerrit:1153937{{!}}Revert &quot;ores-extension: enable extension with revertrisk filter for second batch of wikis&quot;]]
* 09:54 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2172 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77115 and previous config saved to /var/cache/conftool/dbconfig/20250605-095415-fceratto.json
* 09:54 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
* 09:53 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77114 and previous config saved to /var/cache/conftool/dbconfig/20250605-095347-fceratto.json
* 09:53 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 09:53 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 09:52 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2028.codfw.wmnet
* 09:51 marostegui@cumin1002: dbctl commit (dc=all): 'db2158 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77113 and previous config saved to /var/cache/conftool/dbconfig/20250605-095155-root.json
* 09:50 tappof@cumin1002: START - Cookbook sre.hosts.reboot-single for host prometheus1007.eqiad.wmnet
* 09:50 marostegui@cumin1002: dbctl commit (dc=all): 'db2169 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77112 and previous config saved to /var/cache/conftool/dbconfig/20250605-095022-root.json
* 09:44 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2028.codfw.wmnet
* 09:44 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 09:43 claime: Re-enabling CPU/RAM limits on mw-cron - [[phab:T395436|T395436]]
* 09:43 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 09:38 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P77111 and previous config saved to /var/cache/conftool/dbconfig/20250605-093840-fceratto.json
* 09:36 marostegui@cumin1002: dbctl commit (dc=all): 'db2158 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77110 and previous config saved to /var/cache/conftool/dbconfig/20250605-093649-root.json
* 09:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:36 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet
* 09:36 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2027.codfw.wmnet
* 09:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:35 hnowlan: Migrate reading lists API out of restbase for group1 via rest-gateway
* 09:35 marostegui@cumin1002: dbctl commit (dc=all): 'db2169 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77109 and previous config saved to /var/cache/conftool/dbconfig/20250605-093515-root.json
* 09:31 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2158.codfw.wmnet with reason: Maintenance
* 09:31 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2158 [[phab:T395989|T395989]]', diff saved to https://phabricator.wikimedia.org/P77108 and previous config saved to /var/cache/conftool/dbconfig/20250605-093107-marostegui.json
* 09:30 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2027.codfw.wmnet
* 09:26 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet
* 09:25 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;add netflow7002 - jmm@cumin1003&quot;
* 09:24 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2026.codfw.wmnet
* 09:24 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;add netflow7002 - jmm@cumin1003&quot;
* 09:24 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet
* 09:23 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P77107 and previous config saved to /var/cache/conftool/dbconfig/20250605-092333-fceratto.json
* 09:20 marostegui@cumin1002: dbctl commit (dc=all): 'db2169 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77106 and previous config saved to /var/cache/conftool/dbconfig/20250605-092010-root.json
* 09:18 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet
* 09:17 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping2004.codfw.wmnet
* 09:13 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ping2004.codfw.wmnet
* 09:08 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77105 and previous config saved to /var/cache/conftool/dbconfig/20250605-090825-fceratto.json
* 09:07 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:07 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'db2169 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77104 and previous config saved to /var/cache/conftool/dbconfig/20250605-090504-root.json
* 08:58 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2155 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77103 and previous config saved to /var/cache/conftool/dbconfig/20250605-085847-fceratto.json
* 08:58 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
* 08:58 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77102 and previous config saved to /var/cache/conftool/dbconfig/20250605-085820-fceratto.json
* 08:53 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2026.codfw.wmnet
* 08:50 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2025.codfw.wmnet
* 08:50 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet
* 08:50 marostegui@cumin1002: dbctl commit (dc=all): 'db2169 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77101 and previous config saved to /var/cache/conftool/dbconfig/20250605-084959-root.json
* 08:47 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping1004.eqiad.wmnet
* 08:46 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2169.codfw.wmnet with reason: Maintenance
* 08:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2169 [[phab:T395989|T395989]]', diff saved to https://phabricator.wikimedia.org/P77100 and previous config saved to /var/cache/conftool/dbconfig/20250605-084557-marostegui.json
* 08:44 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet
* 08:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ping1004.eqiad.wmnet
* 08:43 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P77099 and previous config saved to /var/cache/conftool/dbconfig/20250605-084313-fceratto.json
* 08:35 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:34 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:32 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:32 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2025.codfw.wmnet
* 08:28 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P77098 and previous config saved to /var/cache/conftool/dbconfig/20250605-082806-fceratto.json
* 08:24 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host netflow7002.magru.wmnet
* 08:24 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host netflow7002.magru.wmnet with OS bookworm
* 08:24 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:24 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 08:12 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77097 and previous config saved to /var/cache/conftool/dbconfig/20250605-081258-fceratto.json
* 08:10 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2024.codfw.wmnet
* 08:09 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2024.codfw.wmnet
* 08:06 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow7002.magru.wmnet with reason: host reimage
* 08:03 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on netflow7002.magru.wmnet with reason: host reimage
* 08:03 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2024.codfw.wmnet
* 08:03 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2147 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77096 and previous config saved to /var/cache/conftool/dbconfig/20250605-080310-fceratto.json
* 08:03 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
* 08:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 08:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 07:56 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2024.codfw.wmnet
* 07:50 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2023.codfw.wmnet
* 07:49 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2023.codfw.wmnet
* 07:44 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:43 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2023.codfw.wmnet
* 07:38 gkyziridis@deploy1003: Sync cancelled.
* 07:36 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host netflow7002.magru.wmnet with OS bookworm
* 07:36 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet
* 07:32 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2022.codfw.wmnet
* 07:32 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2022.codfw.wmnet
* 07:27 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM netflow7002.magru.wmnet - jmm@cumin1003&quot;
* 07:27 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM netflow7002.magru.wmnet - jmm@cumin1003&quot;
* 07:26 jmm@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netflow7002.magru.wmnet on all recursors
* 07:26 jmm@cumin1003: START - Cookbook sre.dns.wipe-cache netflow7002.magru.wmnet on all recursors
* 07:26 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:26 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM netflow7002.magru.wmnet - jmm@cumin1003&quot;
* 07:26 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM netflow7002.magru.wmnet - jmm@cumin1003&quot;
* 07:26 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2022.codfw.wmnet
* 07:23 gkyziridis@deploy1003: gkyziridis: Backport for [[gerrit:1152682{{!}}ores-extension: enable extension with revertrisk filter for second batch of wikis (T395823)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:22 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:22 jmm@cumin1003: START - Cookbook sre.ganeti.makevm for new host netflow7002.magru.wmnet
* 07:21 gkyziridis@deploy1003: Started scap sync-world: Backport for [[gerrit:1152682{{!}}ores-extension: enable extension with revertrisk filter for second batch of wikis (T395823)]]
* 07:19 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2022.codfw.wmnet
* 06:41 marostegui@cumin1002: dbctl commit (dc=all): 'db2180 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77095 and previous config saved to /var/cache/conftool/dbconfig/20250605-064137-root.json
* 06:26 marostegui@cumin1002: dbctl commit (dc=all): 'db2180 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77094 and previous config saved to /var/cache/conftool/dbconfig/20250605-062629-root.json
* 06:19 marostegui: Change datadir on pc8 dbmaint eqiad codfw [[phab:T395983|T395983]]
* 06:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc8 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77093 and previous config saved to /var/cache/conftool/dbconfig/20250605-061929-marostegui.json
* 06:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc8 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77092 and previous config saved to /var/cache/conftool/dbconfig/20250605-061612-marostegui.json
* 06:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2018.codfw.wmnet,pc1018.eqiad.wmnet with reason: Maintenance
* 06:15 marostegui: Change datadir on pc7 dbmaint eqiad codfw [[phab:T395983|T395983]]
* 06:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc7 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77091 and previous config saved to /var/cache/conftool/dbconfig/20250605-061502-marostegui.json
* 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc7 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77090 and previous config saved to /var/cache/conftool/dbconfig/20250605-061200-marostegui.json
* 06:11 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2017.codfw.wmnet,pc1017.eqiad.wmnet with reason: Maintenance
* 06:11 marostegui@cumin1002: dbctl commit (dc=all): 'db2180 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77089 and previous config saved to /var/cache/conftool/dbconfig/20250605-061124-root.json
* 05:57 marostegui: Change datadir on pc6 dbmaint eqiad codfw [[phab:T395983|T395983]]
* 05:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc6 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77088 and previous config saved to /var/cache/conftool/dbconfig/20250605-055655-marostegui.json
* 05:56 marostegui@cumin1002: dbctl commit (dc=all): 'db2180 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77087 and previous config saved to /var/cache/conftool/dbconfig/20250605-055619-root.json
* 05:54 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2016.codfw.wmnet,pc1016.eqiad.wmnet with reason: Maintenance
* 05:54 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc6 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77086 and previous config saved to /var/cache/conftool/dbconfig/20250605-055438-marostegui.json
* 05:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc5 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77085 and previous config saved to /var/cache/conftool/dbconfig/20250605-055349-marostegui.json
* 05:51 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2015.codfw.wmnet,pc1015.eqiad.wmnet with reason: Maintenance
* 05:51 marostegui: Change datadir on pc5 dbmaint eqiad codfw [[phab:T395983|T395983]]
* 05:51 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc5 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77084 and previous config saved to /var/cache/conftool/dbconfig/20250605-055121-marostegui.json
* 05:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc4 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77083 and previous config saved to /var/cache/conftool/dbconfig/20250605-055013-marostegui.json
* 05:48 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2014.codfw.wmnet,pc1014.eqiad.wmnet with reason: Maintenance
* 05:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc4 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77082 and previous config saved to /var/cache/conftool/dbconfig/20250605-054806-marostegui.json
* 05:47 marostegui: Change datadir on pc4 dbmaint eqiad codfw [[phab:T395983|T395983]]
* 05:47 marostegui: Change datadir on pc3 dbmaint eqiad codfw [[phab:T395983|T395983]]
* 05:41 marostegui@cumin1002: dbctl commit (dc=all): 'db2180 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77081 and previous config saved to /var/cache/conftool/dbconfig/20250605-054113-root.json
* 05:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc3 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77080 and previous config saved to /var/cache/conftool/dbconfig/20250605-053647-marostegui.json
* 05:33 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2013.codfw.wmnet,pc1013.eqiad.wmnet with reason: Maintenance
* 05:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc3 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77079 and previous config saved to /var/cache/conftool/dbconfig/20250605-053317-marostegui.json
* 05:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc2 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77078 and previous config saved to /var/cache/conftool/dbconfig/20250605-052934-marostegui.json
* 05:26 marostegui@cumin1002: dbctl commit (dc=all): 'db2180 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77077 and previous config saved to /var/cache/conftool/dbconfig/20250605-052604-root.json
* 05:25 marostegui: Change datadir on pc2 dbmaint eqiad codfw [[phab:T395983|T395983]]
* 05:25 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2012.codfw.wmnet,pc1012.eqiad.wmnet with reason: Maintenance
* 05:24 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc2 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77076 and previous config saved to /var/cache/conftool/dbconfig/20250605-052442-marostegui.json
* 05:20 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2180.codfw.wmnet with reason: Maintenance
* 05:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2180 [[phab:T395989|T395989]]', diff saved to https://phabricator.wikimedia.org/P77075 and previous config saved to /var/cache/conftool/dbconfig/20250605-052003-marostegui.json

== 2025-06-04 ==
* 23:55 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on A:ncredir
* 22:45 brett@cumin2002: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir
* 22:30 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp7016.magru.wmnet
* 22:27 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1185.eqiad.wmnet with OS bullseye
* 22:20 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp7014.magru.wmnet
* 22:18 damilare: SmashPig upgraded from {{Gerrit|d08693e5}} to {{Gerrit|3222a1f3}}
* 22:16 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153725{{!}}Bump cache key version in EventStore (T396075)]] (duration: 13m 54s)
* 22:12 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7016.magru.wmnet
* 22:12 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7015.magru.wmnet
* 22:12 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7015.magru.wmnet
* 22:11 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7015.magru.wmnet
* 22:11 brett: sudo -i cumin 'A:ncredir' 'depool &amp;&amp; apt-get update &amp;&amp; apt-get upgrade -y &amp;&amp; pool' -b1 -s10
* 22:09 ladsgroup@deploy1003: ladsgroup: Continuing with sync
* 22:04 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1153725{{!}}Bump cache key version in EventStore (T396075)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1153725{{!}}Bump cache key version in EventStore (T396075)]]
* 22:02 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7015.magru.wmnet
* 22:02 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7014.magru.wmnet
* 22:02 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp7013.magru.wmnet
* 21:58 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp7012.magru.wmnet
* 21:43 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7013.magru.wmnet
* 21:42 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7011.magru.wmnet
* 21:40 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7012.magru.wmnet
* 21:40 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7010.magru.wmnet
* 21:39 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7010.magru.wmnet
* 21:35 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7010.magru.wmnet
* 21:29 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-reload reloading scholarly_articles on wdqs1023.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/scholarly/20250526/ using stat1011.eqiad.wmnet)
* 21:25 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7011.magru.wmnet
* 21:25 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7010.magru.wmnet
* 21:24 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-reload reloading wikidata_main on wdqs1022.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/main/20250526/ using stat1009.eqiad.wmnet)
* 21:22 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7009.magru.wmnet
* 21:14 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp7008.magru.wmnet
* 21:07 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1185.eqiad.wmnet with OS bullseye
* 21:06 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:05 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7009.magru.wmnet
* 21:05 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7008.magru.wmnet
* 21:04 cjming: end of UTC late backport window
* 21:04 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7007.magru.wmnet
* 21:02 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153689{{!}}SUL3: Retry local login on failure due to invalid/expired login token (T390784)]], [[gerrit:1153690{{!}}SUL3: Retry local login on failure… (follow-ups) (T390784)]], [[gerrit:1153691{{!}}SUL3: Retry local login on failure due to invalid/expired login token (T390784)]], [[gerrit:1153692{{!}}SUL3: Retry local login on failure… (follow-ups) (T390784)]] (d
* 21:01 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 20:55 cjming@deploy1003: matmarex, cjming: Continuing with sync
* 20:55 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:54 cjming@deploy1003: matmarex, cjming: Backport for [[gerrit:1153689{{!}}SUL3: Retry local login on failure due to invalid/expired login token (T390784)]], [[gerrit:1153690{{!}}SUL3: Retry local login on failure… (follow-ups) (T390784)]], [[gerrit:1153691{{!}}SUL3: Retry local login on failure due to invalid/expired login token (T390784)]], [[gerrit:1153692{{!}}SUL3: Retry local login on failure… (follow-ups) (T390784)]] synced to
* 20:51 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1153689{{!}}SUL3: Retry local login on failure due to invalid/expired login token (T390784)]], [[gerrit:1153690{{!}}SUL3: Retry local login on failure… (follow-ups) (T390784)]], [[gerrit:1153691{{!}}SUL3: Retry local login on failure due to invalid/expired login token (T390784)]], [[gerrit:1153692{{!}}SUL3: Retry local login on failure… (follow-ups) (T390784)]]
* 20:51 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 20:50 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7006.magru.wmnet
* 20:46 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7007.magru.wmnet
* 20:44 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp7005.magru.wmnet
* 20:40 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7006.magru.wmnet
* 20:38 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153686{{!}}Treat File::getShortDesc() as possibly unsafe HTML (T395834)]], [[gerrit:1153687{{!}}Treat File::getShortDesc() as possibly unsafe HTML (T395834)]] (duration: 15m 37s)
* 20:37 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7004.magru.wmnet
* 20:31 cjming@deploy1003: cjming, matmarex: Continuing with sync
* 20:26 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7005.magru.wmnet
* 20:26 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7004.magru.wmnet
* 20:25 cjming@deploy1003: cjming, matmarex: Backport for [[gerrit:1153686{{!}}Treat File::getShortDesc() as possibly unsafe HTML (T395834)]], [[gerrit:1153687{{!}}Treat File::getShortDesc() as possibly unsafe HTML (T395834)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:23 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1153686{{!}}Treat File::getShortDesc() as possibly unsafe HTML (T395834)]], [[gerrit:1153687{{!}}Treat File::getShortDesc() as possibly unsafe HTML (T395834)]]
* 20:20 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7003.magru.wmnet
* 20:18 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7002.magru.wmnet
* 20:15 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153673{{!}}beta cluster: Set $wgOATHAuthAccountPrefix (T396061)]] (duration: 10m 13s)
* 20:10 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7003.magru.wmnet
* 20:09 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7002.magru.wmnet
* 20:08 cjming@deploy1003: lucaswerkmeister, cjming: Continuing with sync
* 20:07 cjming@deploy1003: lucaswerkmeister, cjming: Backport for [[gerrit:1153673{{!}}beta cluster: Set $wgOATHAuthAccountPrefix (T396061)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:05 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1153673{{!}}beta cluster: Set $wgOATHAuthAccountPrefix (T396061)]]
* 20:03 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp3072.esams.wmnet
* 19:54 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp3072.esams.wmnet
* 19:42 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp7001.magru.wmnet
* 19:36 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 19:31 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp7001.magru.wmnet
* 19:23 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 19:22 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 19:13 sukhe@dns1004: END - running authdns-update
* 19:12 sukhe@dns1004: START - running authdns-update
* 19:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.wikimedia.org [reason: repooling after reboot]
* 19:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.magru.wmnet [reason: repooling after reboot]
* 19:11 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns7002.wikimedia.org
* 19:11 sukhe@cumin1002: START - Cookbook sre.hosts.remove-downtime for dns7002.wikimedia.org
* 19:10 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on A:dnsbox and A:magru and (A:dnsbox)
* 19:10 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot finished rebooting dns7002.wikimedia.org
* 19:10 bvibber@deploy1003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 19:10 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 19:10 bvibber@deploy1003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 19:10 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153679{{!}}CustomBlockedDomainStorage::fetchConfig: Cast LinkTarget to a Title for RevisionStore (T396056)]] (duration: 12m 27s)
* 19:10 bvibber@deploy1003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 19:09 bvibber@deploy1003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 19:09 bvibber@deploy1003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:08 bvibber@deploy1003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:03 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync
* 19:00 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 19:00 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 19:00 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 18:59 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 18:59 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1153679{{!}}CustomBlockedDomainStorage::fetchConfig: Cast LinkTarget to a Title for RevisionStore (T396056)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:57 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1153679{{!}}CustomBlockedDomainStorage::fetchConfig: Cast LinkTarget to a Title for RevisionStore (T396056)]]
* 18:55 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot begin reboot of dns7002.wikimedia.org
* 18:51 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7001.*
* 18:49 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 18:45 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot finished rebooting dns7001.wikimedia.org
* 18:37 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7001.*
* 18:37 brett: depooling cp7001 for CPU stress testing and temperature effects ([[phab:T373993|T373993]])
* 18:29 sukhe@cumin1002: cookbooks.sre.dns.roll-reboot begin reboot of dns7001.wikimedia.org
* 18:29 sukhe@cumin1002: START - Cookbook sre.dns.roll-reboot rolling reboot on A:dnsbox and A:magru and (A:dnsbox)
* 18:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 18:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 18:18 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.45.0-wmf.4  refs [[phab:T392174|T392174]]
* 18:16 damilare: SmashPig upgraded from {{Gerrit|a99f2265}} to {{Gerrit|d08693e5}}
* 18:15 sukhe: puppet re-enabled on A:cp and finished rolling out removal of ats-be from single backend cp nodes: [[phab:T288106|T288106]]
* 18:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 18:13 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 18:03 bvibber@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153662{{!}}Update Charts so they can render from data-mw-charts as well as data-charts (T395462)]], [[gerrit:1153663{{!}}Update Charts so they can render from data-mw-charts as well as data-charts (T395462)]] (duration: 10m 05s)
* 17:56 bvibber@deploy1003: bvibber: Continuing with sync
* 17:55 bvibber@deploy1003: bvibber: Backport for [[gerrit:1153662{{!}}Update Charts so they can render from data-mw-charts as well as data-charts (T395462)]], [[gerrit:1153663{{!}}Update Charts so they can render from data-mw-charts as well as data-charts (T395462)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:53 bvibber@deploy1003: Started scap sync-world: Backport for [[gerrit:1153662{{!}}Update Charts so they can render from data-mw-charts as well as data-charts (T395462)]], [[gerrit:1153663{{!}}Update Charts so they can render from data-mw-charts as well as data-charts (T395462)]]
* 17:50 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: sync
* 17:49 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: sync
* 17:48 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 17:48 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 17:36 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 17:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 17:15 cgoubert@deploy1003: Finished scap sync-world: {{Gerrit|1153647}}: mediawiki: Fix captcha configmap structure - [[phab:T388531|T388531]] (duration: 02m 39s)
* 17:13 cgoubert@deploy1003: Started scap sync-world: {{Gerrit|1153647}}: mediawiki: Fix captcha configmap structure - [[phab:T388531|T388531]]
* 16:41 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1185.eqiad.wmnet with OS bullseye
* 16:33 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 16:32 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 16:30 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1186
* 16:30 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1186
* 16:27 stevemunene@cumin1002: END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1165.eqiad.wmnet
* 16:27 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1165.eqiad.wmnet
* 16:22 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:22 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  an-worker1186 - vriley@cumin1002&quot;
* 16:22 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update  mgmt  an-worker1186 - vriley@cumin1002&quot;
* 16:19 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) for hosts an-worker1164.eqiad.wmnet
* 16:18 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1164.eqiad.wmnet
* 16:18 stevemunene@cumin1002: END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1165.eqiad.wmnet
* 16:16 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1165.eqiad.wmnet
* 16:14 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 16:14 stevemunene@cumin1002: END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1164.eqiad.wmnet
* 16:13 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1164.eqiad.wmnet
* 16:13 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 16:12 stevemunene@cumin1002: END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1164.eqiad.wmnet
* 16:12 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum and not A:magru and A:durum
* 16:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 16:11 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough and not A:magru and A:wikidough
* 16:10 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1164.eqiad.wmnet
* 16:07 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1163.eqiad.wmnet
* 16:04 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 16:04 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 16:03 stevemunene@cumin1002: START - Cookbook sre.hosts.reboot-single for host an-worker1163.eqiad.wmnet
* 16:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77074 and previous config saved to /var/cache/conftool/dbconfig/20250604-160120-fceratto.json
* 15:56 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 15:56 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 15:46 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P77073 and previous config saved to /var/cache/conftool/dbconfig/20250604-154611-fceratto.json
* 15:42 tappof@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2005.codfw.wmnet
* 15:31 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P77072 and previous config saved to /var/cache/conftool/dbconfig/20250604-153104-fceratto.json
* 15:30 tappof@cumin1002: START - Cookbook sre.hosts.reboot-single for host prometheus2005.codfw.wmnet
* 15:29 tappof@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1005.eqiad.wmnet
* 15:24 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153646{{!}}Set wgCheckUserDisableCheckUserAPI to false on loginwiki (T396010)]] (duration: 10m 03s)
* 15:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1185.eqiad.wmnet with OS bullseye
* 15:20 tappof@cumin1002: START - Cookbook sre.hosts.reboot-single for host prometheus1005.eqiad.wmnet
* 15:18 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:18 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 15:18 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 15:17 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync
* 15:16 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1153646{{!}}Set wgCheckUserDisableCheckUserAPI to false on loginwiki (T396010)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:15 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77071 and previous config saved to /var/cache/conftool/dbconfig/20250604-151556-fceratto.json
* 15:15 sukhe@cumin1002: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum and not A:magru and A:durum
* 15:14 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1153646{{!}}Set wgCheckUserDisableCheckUserAPI to false on loginwiki (T396010)]]
* 15:14 sukhe@cumin1002: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough and not A:magru and A:wikidough
* 15:14 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-worker1185.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 15:08 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 15:07 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77070 and previous config saved to /var/cache/conftool/dbconfig/20250604-150740-fceratto.json
* 15:07 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 15:07 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77069 and previous config saved to /var/cache/conftool/dbconfig/20250604-150716-fceratto.json
* 15:05 jiji@deploy1003: Finished scap sync-world: [[phab:T276994|T276994]]: Chart bump, noop (duration: 02m 52s)
* 15:04 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol2010-dev.codfw.wmnet with OS bookworm
* 15:04 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:02 jiji@deploy1003: Started scap sync-world: [[phab:T276994|T276994]]: Chart bump, noop
* 15:00 cmooney@cumin1002: START - Cookbook sre.dns.netbox
* 15:00 cmooney@cumin1002: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97)
* 15:00 cmooney@cumin1002: START - Cookbook sre.dns.netbox
* 15:00 stevemunene@cumin1002: END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1163.eqiad.wmnet
* 14:58 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1163.eqiad.wmnet
* 14:58 moritzm: installing Linux 5.10.237 on Bullseye hosts
* 14:55 cmooney@dns2005: END - running authdns-update
* 14:54 cmooney@dns2005: START - running authdns-update
* 14:52 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P77068 and previous config saved to /var/cache/conftool/dbconfig/20250604-145209-fceratto.json
* 14:50 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:50 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove DNS entries for IPs used in Nokia test lab codfw - cmooney@cumin1002&quot;
* 14:49 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove DNS entries for IPs used in Nokia test lab codfw - cmooney@cumin1002&quot;
* 14:49 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1094.eqiad.wmnet with OS bullseye
* 14:49 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002&quot;
* 14:49 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002&quot;
* 14:47 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2020.codfw.wmnet
* 14:47 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2020.codfw.wmnet
* 14:46 vriley@cumin1002: START - Cookbook sre.hosts.provision for host an-worker1185.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 14:46 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol2010-dev.codfw.wmnet with reason: host reimage
* 14:46 cmooney@cumin1002: START - Cookbook sre.dns.netbox
* 14:46 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1095.eqiad.wmnet with OS bullseye
* 14:46 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002&quot;
* 14:44 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002&quot;
* 14:43 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol2010-dev.codfw.wmnet with reason: host reimage
* 14:41 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2020.codfw.wmnet
* 14:38 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 14:37 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 14:37 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P77067 and previous config saved to /var/cache/conftool/dbconfig/20250604-143702-fceratto.json
* 14:36 cgoubert@deploy1003: Finished scap sync-world: {{Gerrit|1153634}}: mediawiki: Fix captcha wordlists path - [[phab:T388531|T388531]] (duration: 02m 33s)
* 14:33 cgoubert@deploy1003: Started scap sync-world: {{Gerrit|1153634}}: mediawiki: Fix captcha wordlists path - [[phab:T388531|T388531]]
* 14:32 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 14:32 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 14:31 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2020.codfw.wmnet
* 14:31 cgoubert@deploy1003: Finished scap sync-world: {{Gerrit|1153634}}: mediawiki: Fix captcha wordlists path - [[phab:T388531|T388531]] (duration: 02m 24s)
* 14:29 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2019.codfw.wmnet
* 14:29 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2019.codfw.wmnet
* 14:28 cgoubert@deploy1003: Started scap sync-world: {{Gerrit|1153634}}: mediawiki: Fix captcha wordlists path - [[phab:T388531|T388531]]
* 14:28 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 14:27 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 14:27 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 14:26 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 14:26 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 14:25 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 14:24 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:24 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts doh7001.wikimedia.org
* 14:24 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:24 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: doh7001.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin1002&quot;
* 14:23 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: doh7001.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin1002&quot;
* 14:23 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:23 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2019.codfw.wmnet
* 14:21 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77066 and previous config saved to /var/cache/conftool/dbconfig/20250604-142155-fceratto.json
* 14:21 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:21 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:20 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcontrol2010-dev.codfw.wmnet with OS bookworm
* 14:19 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts durum7001.magru.wmnet
* 14:19 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:19 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: durum7001.magru.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1002&quot;
* 14:19 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: durum7001.magru.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1002&quot;
* 14:19 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol2010-dev.codfw.wmnet with OS bullseye
* 14:19 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002&quot;
* 14:17 sukhe@cumin1002: START - Cookbook sre.dns.netbox
* 14:16 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet
* 14:14 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002&quot;
* 14:13 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2019.codfw.wmnet
* 14:12 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77065 and previous config saved to /var/cache/conftool/dbconfig/20250604-141238-fceratto.json
* 14:12 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 14:12 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77064 and previous config saved to /var/cache/conftool/dbconfig/20250604-141213-fceratto.json
* 14:11 sukhe@cumin1002: START - Cookbook sre.dns.netbox
* 14:11 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet
* 14:08 sukhe: decommissioning doh7001 and durum7001: [[phab:T396015|T396015]]
* 14:07 sukhe@cumin1002: START - Cookbook sre.hosts.decommission for hosts doh7001.wikimedia.org
* 14:07 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:07 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:07 sukhe@cumin1002: START - Cookbook sre.hosts.decommission for hosts durum7001.magru.wmnet
* 14:06 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:06 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:05 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:05 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:04 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 14:04 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.admin config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 14:02 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 14:02 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 14:01 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1094.eqiad.wmnet with reason: host reimage
* 13:58 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol2010-dev.codfw.wmnet with reason: host reimage
* 13:57 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P77062 and previous config saved to /var/cache/conftool/dbconfig/20250604-135706-fceratto.json
* 13:56 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1094.eqiad.wmnet with reason: host reimage
* 13:54 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol2010-dev.codfw.wmnet with reason: host reimage
* 13:51 claime: Manual run of generatecaptcha on mw-cron, no delete - [[phab:T388531|T388531]]
* 13:51 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 13:50 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 13:49 sukhe: sudo cumin -b1 -s15 'A:cp' 'run-puppet-agent --enable &quot;merging CR 1114074&quot;': [[phab:T288106|T288106]]
* 13:48 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 13:48 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.admin config_reloading P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs1013.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 13:47 tappof@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host grafana2001.codfw.wmnet
* 13:46 sukhe: forcing ats-backend-restart on cp1104
* 13:43 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1095.eqiad.wmnet with reason: host reimage
* 13:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2217 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77061 and previous config saved to /var/cache/conftool/dbconfig/20250604-134336-root.json
* 13:43 tappof@cumin1002: START - Cookbook sre.hosts.reboot-single for host grafana2001.codfw.wmnet
* 13:41 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P77060 and previous config saved to /var/cache/conftool/dbconfig/20250604-134158-fceratto.json
* 13:41 tappof@cumin1002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host grafana2001.codfw.wmnet
* 13:41 tappof@cumin1002: START - Cookbook sre.hosts.reboot-single for host grafana2001.codfw.wmnet
* 13:40 tappof@cumin1002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host grafana2001.codfw.wmnet
* 13:40 tappof@cumin1002: START - Cookbook sre.hosts.reboot-single for host grafana2001.codfw.wmnet
* 13:40 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153623{{!}}IS: Undo turning on wgTemplateDataEnableCategoryBrowser for mw.org (T377975)]] (duration: 09m 57s)
* 13:39 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1095.eqiad.wmnet with reason: host reimage
* 13:38 tappof@cumin1002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host grafana2001.codfw.wmnet
* 13:38 tappof@cumin1002: START - Cookbook sre.hosts.reboot-single for host grafana2001.codfw.wmnet
* 13:37 sukhe: forcing agent run on cp2037 (non-single BE node): CR {{Gerrit|1114074}}
* 13:37 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1094.eqiad.wmnet with OS bullseye
* 13:33 samtar@deploy1003: samtar: Continuing with sync
* 13:32 samtar@deploy1003: samtar: Backport for [[gerrit:1153623{{!}}IS: Undo turning on wgTemplateDataEnableCategoryBrowser for mw.org (T377975)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:30 sukhe: forcing agent run on cp7001 (single BE node): CR {{Gerrit|1114074}}
* 13:30 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1153623{{!}}IS: Undo turning on wgTemplateDataEnableCategoryBrowser for mw.org (T377975)]]
* 13:29 sukhe: forcing agent run on cp6015: CR {{Gerrit|1114074}}
* 13:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2217 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77059 and previous config saved to /var/cache/conftool/dbconfig/20250604-132829-root.json
* 13:27 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1013.eqiad.wmnet
* 13:27 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs1013.eqiad.wmnet
* 13:26 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77058 and previous config saved to /var/cache/conftool/dbconfig/20250604-132648-fceratto.json
* 13:23 sukhe: starting removal of ats-be service from eqiad, eqsin, esams, magru, ulsfo: [[phab:T288106|T288106]]
* 13:21 sukhe: sudo cumin 'A:cp' 'disable-puppet &quot;merging CR 1114074&quot;'
* 13:20 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1095.eqiad.wmnet with OS bullseye
* 13:18 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77057 and previous config saved to /var/cache/conftool/dbconfig/20250604-131852-fceratto.json
* 13:18 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 13:18 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77056 and previous config saved to /var/cache/conftool/dbconfig/20250604-131827-fceratto.json
* 13:14 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1146628{{!}}release CampaignEvents to cbk-zam wiki (T393604)]], [[gerrit:1153385{{!}}Bump portals to the 2025-06-02 09:23:11+00:00 build (T128546)]], [[gerrit:1151781{{!}}build: Rename the rarely-used 'typos' script to 'checkTypos']], [[gerrit:1151751{{!}}Drop Chart roll-out dblists, no longer needed (T383079)]] (duration: 10m 29s)
* 13:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2217 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77055 and previous config saved to /var/cache/conftool/dbconfig/20250604-131323-root.json
* 13:11 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply
* 13:11 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply
* 13:08 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
* 13:07 jforrester@deploy1003: jforrester, mhorsey: Continuing with sync
* 13:06 jforrester@deploy1003: jforrester, mhorsey: Backport for [[gerrit:1146628{{!}}release CampaignEvents to cbk-zam wiki (T393604)]], [[gerrit:1153385{{!}}Bump portals to the 2025-06-02 09:23:11+00:00 build (T128546)]], [[gerrit:1151781{{!}}build: Rename the rarely-used 'typos' script to 'checkTypos']], [[gerrit:1151751{{!}}Drop Chart roll-out dblists, no longer needed (T383079)]] synced to the testservers (see https://wikitech.wikimedia
* 13:04 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti7001.magru.wmnet to cluster magru03 and group B
* 13:04 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1146628{{!}}release CampaignEvents to cbk-zam wiki (T393604)]], [[gerrit:1153385{{!}}Bump portals to the 2025-06-02 09:23:11+00:00 build (T128546)]], [[gerrit:1151781{{!}}build: Rename the rarely-used 'typos' script to 'checkTypos']], [[gerrit:1151751{{!}}Drop Chart roll-out dblists, no longer needed (T383079)]]
* 13:03 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P77054 and previous config saved to /var/cache/conftool/dbconfig/20250604-130319-fceratto.json
* 13:03 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti7001.magru.wmnet to cluster magru03 and group B
* 13:02 sbassett@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 13:02 sbassett@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 13:02 sbassett@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 13:01 sbassett@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 13:01 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7001.magru.wmnet
* 12:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2217 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77053 and previous config saved to /var/cache/conftool/dbconfig/20250604-125817-root.json
* 12:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2217 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77051 and previous config saved to /var/cache/conftool/dbconfig/20250604-124311-root.json
* 12:43 andrew@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudgw1004.eqiad.wmnet
* 12:42 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw1003.eqiad.wmnet
* 12:39 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
* 12:39 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti7001.magru.wmnet with OS bookworm
* 12:39 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
* 12:37 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1095.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 12:36 moritzm: installing modsecurity-apache security updates
* 12:36 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1094.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 12:35 andrew@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudgw1003.eqiad.wmnet
* 12:35 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:35 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: added mgmt for ms-be1094/95 - jclark@cumin1002&quot;
* 12:35 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: added mgmt for ms-be1094/95 - jclark@cumin1002&quot;
* 12:34 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcontrol2010-dev.codfw.wmnet with OS bullseye
* 12:33 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77050 and previous config saved to /var/cache/conftool/dbconfig/20250604-123304-fceratto.json
* 12:32 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2193 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77049 and previous config saved to /var/cache/conftool/dbconfig/20250604-122948-root.json
* {{safesubst:SAL entry|1=12:28 reedy@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153591{{!}}GenerateFancyCaptchas: Handle captcha.py not generating any captchas, but not erroring (T388531)]], [[gerrit:1153592{{!}}captcha.py: Expand variables and user in filenames (T395810)]], [[gerrit:1153593{{!}}captcha.py: Check if output dir exists, and attempt to create it (else error) (T395804)]], [[gerrit:1153595{{!}}captcha.py: Bail out if no words were rea}}
* 12:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2217 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77048 and previous config saved to /var/cache/conftool/dbconfig/20250604-122806-root.json
* 12:27 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:27 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cloudcontrol2010-dev.codfw.wmnet on all recursors
* 12:27 cmooney@cumin1002: START - Cookbook sre.dns.wipe-cache cloudcontrol2010-dev.codfw.wmnet on all recursors
* 12:26 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cloudcephosd2010-dev.codfw.wmnet on all recursors
* 12:26 cmooney@cumin1002: START - Cookbook sre.dns.wipe-cache cloudcephosd2010-dev.codfw.wmnet on all recursors
* 12:25 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 12:25 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:25 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: fix entries for cloudcontrol2010-dev which had been added on wrong vlan - cmooney@cumin1002&quot;
* 12:25 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: fix entries for cloudcontrol2010-dev which had been added on wrong vlan - cmooney@cumin1002&quot;
* 12:24 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1203 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77047 and previous config saved to /var/cache/conftool/dbconfig/20250604-122436-fceratto.json
* 12:24 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance
* 12:24 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77046 and previous config saved to /var/cache/conftool/dbconfig/20250604-122411-fceratto.json
* 12:23 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2217 [[phab:T395989|T395989]]', diff saved to https://phabricator.wikimedia.org/P77045 and previous config saved to /var/cache/conftool/dbconfig/20250604-122303-marostegui.json
* 12:22 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2217.codfw.wmnet with reason: Maintenance
* 12:21 cmooney@cumin1002: START - Cookbook sre.dns.netbox
* 12:21 reedy@deploy1003: reedy: Continuing with sync
* 12:21 cmooney@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* {{safesubst:SAL entry|1=12:20 reedy@deploy1003: reedy: Backport for [[gerrit:1153591{{!}}GenerateFancyCaptchas: Handle captcha.py not generating any captchas, but not erroring (T388531)]], [[gerrit:1153592{{!}}captcha.py: Expand variables and user in filenames (T395810)]], [[gerrit:1153593{{!}}captcha.py: Check if output dir exists, and attempt to create it (else error) (T395804)]], [[gerrit:1153595{{!}}captcha.py: Bail out if no words were read from wordlist (T3}}
* 12:18 cmooney@cumin1002: START - Cookbook sre.dns.netbox
* {{safesubst:SAL entry|1=12:18 reedy@deploy1003: Started scap sync-world: Backport for [[gerrit:1153591{{!}}GenerateFancyCaptchas: Handle captcha.py not generating any captchas, but not erroring (T388531)]], [[gerrit:1153592{{!}}captcha.py: Expand variables and user in filenames (T395810)]], [[gerrit:1153593{{!}}captcha.py: Check if output dir exists, and attempt to create it (else error) (T395804)]], [[gerrit:1153595{{!}}captcha.py: Bail out if no words were read}}
* 12:16 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti7001.magru.wmnet with reason: host reimage
* 12:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2193 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77044 and previous config saved to /var/cache/conftool/dbconfig/20250604-121442-root.json
* 12:12 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti7001.magru.wmnet with reason: host reimage
* 12:09 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P77043 and previous config saved to /var/cache/conftool/dbconfig/20250604-120904-fceratto.json
* 11:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2193 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77041 and previous config saved to /var/cache/conftool/dbconfig/20250604-115936-root.json
* 11:58 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153581{{!}}IS/IS-labs: Enable TemplateDiscovery flags for mediawikiwiki (T377975)]] (duration: 12m 28s)
* 11:55 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts krb1001.eqiad.wmnet
* 11:55 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:55 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: krb1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 11:53 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P77040 and previous config saved to /var/cache/conftool/dbconfig/20250604-115357-fceratto.json
* 11:51 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: krb1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003&quot;
* 11:51 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti7001.magru.wmnet with OS bookworm
* 11:51 samtar@deploy1003: samtar: Continuing with sync
* 11:47 samtar@deploy1003: samtar: Backport for [[gerrit:1153581{{!}}IS/IS-labs: Enable TemplateDiscovery flags for mediawikiwiki (T377975)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:45 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1153581{{!}}IS/IS-labs: Enable TemplateDiscovery flags for mediawikiwiki (T377975)]]
* 11:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2193 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77038 and previous config saved to /var/cache/conftool/dbconfig/20250604-114430-root.json
* 11:38 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77037 and previous config saved to /var/cache/conftool/dbconfig/20250604-113849-fceratto.json
* 11:38 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 11:35 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 11:35 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' .
* 11:35 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 11:34 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'ores-legacy' for release 'main' .
* 11:34 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4008.ulsfo.wmnet
* 11:34 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet
* 11:33 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 11:32 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 11:32 jmm@cumin1003: START - Cookbook sre.hosts.decommission for hosts krb1001.eqiad.wmnet
* 11:31 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 11:31 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply
* 11:30 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77036 and previous config saved to /var/cache/conftool/dbconfig/20250604-113030-fceratto.json
* 11:30 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 11:30 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77035 and previous config saved to /var/cache/conftool/dbconfig/20250604-113005-fceratto.json
* 11:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2193 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77034 and previous config saved to /var/cache/conftool/dbconfig/20250604-112923-root.json
* 11:27 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet
* 11:25 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 11:25 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply
* 11:22 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 11:22 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply
* 11:20 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4008.ulsfo.wmnet
* 11:14 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P77033 and previous config saved to /var/cache/conftool/dbconfig/20250604-111457-fceratto.json
* 11:14 claime: Deployed k8s-controller-sidecars version 1.0.2-3
* 11:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2193 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77032 and previous config saved to /var/cache/conftool/dbconfig/20250604-111418-root.json
* 11:10 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2193.codfw.wmnet with reason: Maintenance
* 11:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2193 [[phab:T395989|T395989]]', diff saved to https://phabricator.wikimedia.org/P77031 and previous config saved to /var/cache/conftool/dbconfig/20250604-110955-marostegui.json
* 11:06 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 11:06 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 11:06 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 11:05 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 11:05 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 11:04 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 11:04 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 11:03 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 10:59 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P77030 and previous config saved to /var/cache/conftool/dbconfig/20250604-105950-fceratto.json
* 10:52 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti7001.magru.wmnet to cluster magru03 and group B
* 10:51 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti7001.magru.wmnet to cluster magru03 and group B
* 10:45 akosiaris@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:44 akosiaris@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:44 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77029 and previous config saved to /var/cache/conftool/dbconfig/20250604-104443-fceratto.json
* 10:38 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti7001.magru.wmnet to cluster magru03 and group B
* 10:37 jmm@cumin1003: START - Cookbook sre.ganeti.addnode for new host ganeti7001.magru.wmnet to cluster magru03 and group B
* 10:36 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1178 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77028 and previous config saved to /var/cache/conftool/dbconfig/20250604-103629-fceratto.json
* 10:36 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance
* 10:36 moritzm: failover ganeti master in ulsfo to ganeti4005
* 10:36 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77027 and previous config saved to /var/cache/conftool/dbconfig/20250604-103604-fceratto.json
* 10:35 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4007.ulsfo.wmnet
* 10:35 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7001.magru.wmnet
* 10:35 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4007.ulsfo.wmnet
* 10:32 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77026 and previous config saved to /var/cache/conftool/dbconfig/20250604-103233-root.json
* 10:29 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti4007.ulsfo.wmnet
* 10:25 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4007.ulsfo.wmnet
* 10:25 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti7001.magru.wmnet
* 10:23 marostegui@cumin1002: dbctl commit (dc=all): 'es2041 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77025 and previous config saved to /var/cache/conftool/dbconfig/20250604-102351-root.json
* 10:23 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4006.ulsfo.wmnet
* 10:23 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4006.ulsfo.wmnet
* 10:20 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P77024 and previous config saved to /var/cache/conftool/dbconfig/20250604-102056-fceratto.json
* 10:17 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77023 and previous config saved to /var/cache/conftool/dbconfig/20250604-101728-root.json
* 10:17 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti4006.ulsfo.wmnet
* 10:09 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4006.ulsfo.wmnet
* 10:08 marostegui@cumin1002: dbctl commit (dc=all): 'es2041 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77022 and previous config saved to /var/cache/conftool/dbconfig/20250604-100846-root.json
* 10:05 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P77021 and previous config saved to /var/cache/conftool/dbconfig/20250604-100547-fceratto.json
* 10:05 vgutierrez@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on lvs1013.eqiad.wmnet with reason: switching to katran
* 10:04 vgutierrez: upload liberica 0.15 to bookwork-wikimedia (apt.wm.o) - [[phab:T395228|T395228]]
* 10:02 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77020 and previous config saved to /var/cache/conftool/dbconfig/20250604-100222-root.json
* 10:00 vgutierrez: depool lvs1013 before switching to katran - [[phab:T395228|T395228]]
* 09:59 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:59 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:55 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 09:55 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 09:53 marostegui@cumin1002: dbctl commit (dc=all): 'es2041 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77018 and previous config saved to /var/cache/conftool/dbconfig/20250604-095340-root.json
* 09:52 akosiaris@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 09:52 akosiaris: re-deploy changeprop-jobqueue to set the Host HTTP header for submission of all jobs. [[phab:T395451|T395451]]
* 09:52 akosiaris@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 09:50 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77017 and previous config saved to /var/cache/conftool/dbconfig/20250604-095041-fceratto.json
* 09:47 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77016 and previous config saved to /var/cache/conftool/dbconfig/20250604-094715-root.json
* 09:46 akosiaris@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
* 09:46 akosiaris: [[phab:T395451|T395451]] deploy mw-jobrunner hot patch for VirtualHost selection, testing out that the single version change will work this time around.
* 09:46 akosiaris@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
* 09:42 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1177 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77015 and previous config saved to /var/cache/conftool/dbconfig/20250604-094217-fceratto.json
* 09:42 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4005.ulsfo.wmnet
* 09:42 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance
* 09:42 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4005.ulsfo.wmnet
* 09:41 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P77014 and previous config saved to /var/cache/conftool/dbconfig/20250604-094152-fceratto.json
* 09:40 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti7001.magru.wmnet with OS bookworm
* 09:38 marostegui@cumin1002: dbctl commit (dc=all): 'es2041 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77013 and previous config saved to /var/cache/conftool/dbconfig/20250604-093835-root.json
* 09:37 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 09:37 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 09:36 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti4005.ulsfo.wmnet
* 09:33 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4005.ulsfo.wmnet
* 09:32 marostegui@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P77011 and previous config saved to /var/cache/conftool/dbconfig/20250604-093251-root.json
* 09:32 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77010 and previous config saved to /var/cache/conftool/dbconfig/20250604-093209-root.json
* 09:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc1 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77009 and previous config saved to /var/cache/conftool/dbconfig/20250604-092819-marostegui.json
* 09:27 marostegui: Move datadir on pc1011 dbmaint pc1 eqiad [[phab:T395983|T395983]]
* 09:26 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P77008 and previous config saved to /var/cache/conftool/dbconfig/20250604-092645-fceratto.json
* 09:24 taavi@cumin1002: conftool action : set/weight=100:pooled=yes; selector: name=clouddb1020.eqiad.wmnet,service=x3
* 09:23 marostegui@cumin1002: dbctl commit (dc=all): 'es2041 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77007 and previous config saved to /var/cache/conftool/dbconfig/20250604-092328-root.json
* 09:20 marostegui: Move datadir on pc2011 dbmaint pc1 codfw [[phab:T395983|T395983]]
* 09:19 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2011.codfw.wmnet,pc1011.eqiad.wmnet with reason: Maintenance
* 09:19 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc1 [[phab:T395983|T395983]]', diff saved to https://phabricator.wikimedia.org/P77006 and previous config saved to /var/cache/conftool/dbconfig/20250604-091921-marostegui.json
* 09:17 marostegui@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77005 and previous config saved to /var/cache/conftool/dbconfig/20250604-091745-root.json
* 09:17 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77004 and previous config saved to /var/cache/conftool/dbconfig/20250604-091704-root.json
* 09:16 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 09:16 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti7001.magru.wmnet with reason: host reimage
* 09:16 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 09:15 akosiaris@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 09:15 akosiaris@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 09:15 akosiaris: [[phab:T395451|T395451]] rollback the host header addition, this is erroring out, returning 404s.
* 09:14 akosiaris: [[phab:T395451|T395451]] rollback the host header addition, this is erroring out, returning 3xx.
* 09:14 akosiaris@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 09:14 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 09:14 akosiaris@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 09:12 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti7001.magru.wmnet with reason: host reimage
* 09:11 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P77003 and previous config saved to /var/cache/conftool/dbconfig/20250604-091138-fceratto.json
* 09:11 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 09:11 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es2044.codfw.wmnet with reason: Maintenance
* 09:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2044', diff saved to https://phabricator.wikimedia.org/P77002 and previous config saved to /var/cache/conftool/dbconfig/20250604-091041-marostegui.json
* 09:10 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 09:10 moritzm: installing qemu bugfix updates from Bookworm point release
* 09:08 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f1-codfw
* 09:08 marostegui@cumin1002: dbctl commit (dc=all): 'es2041 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77001 and previous config saved to /var/cache/conftool/dbconfig/20250604-090823-root.json
* 09:06 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-f1-codfw
* 09:05 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 09:03 akosiaris@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 09:02 akosiaris@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 09:02 marostegui@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77000 and previous config saved to /var/cache/conftool/dbconfig/20250604-090240-root.json
* 09:02 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2041', diff saved to https://phabricator.wikimedia.org/P76999 and previous config saved to /var/cache/conftool/dbconfig/20250604-090226-marostegui.json
* 09:02 akosiaris@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 09:02 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es2041.codfw.wmnet with reason: Maintenance
* 09:01 akosiaris@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 08:57 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 08:57 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 08:56 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76998 and previous config saved to /var/cache/conftool/dbconfig/20250604-085630-fceratto.json
* 08:53 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 08:53 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 08:51 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ganeti7001.magru.wmnet with OS bookworm
* 08:48 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1172 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76997 and previous config saved to /var/cache/conftool/dbconfig/20250604-084819-fceratto.json
* 08:48 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
* 08:47 marostegui@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76996 and previous config saved to /var/cache/conftool/dbconfig/20250604-084735-root.json
* 08:42 akosiaris: deploy changeprop-jobqueue to set the Host HTTP header for submission of all jobs. [[phab:T395451|T395451]]
* 08:42 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 08:42 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76995 and previous config saved to /var/cache/conftool/dbconfig/20250604-084231-fceratto.json
* 08:42 akosiaris@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 08:41 akosiaris@deploy1003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 08:38 moritzm: removing ganeti7001 from magru01 cluster [[phab:T394263|T394263]]
* 08:38 marostegui: Change s6 eqiad dbmaint to SBR [[phab:T383795|T383795]]
* 08:38 akosiaris: revoke and clean helm-charts.discovery.wmnet old cergen cert from puppetmaster1001
* 08:32 marostegui@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76994 and previous config saved to /var/cache/conftool/dbconfig/20250604-083229-root.json
* 08:28 marostegui: Change s6 codfw dbmaint to SBR [[phab:T383795|T383795]]
* 08:27 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P76993 and previous config saved to /var/cache/conftool/dbconfig/20250604-082725-fceratto.json
* 08:17 marostegui@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76992 and previous config saved to /var/cache/conftool/dbconfig/20250604-081725-root.json
* 08:14 moritzm: removing atlas7001 from magru01 cluster [[phab:T394263|T394263]]
* 08:12 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P76991 and previous config saved to /var/cache/conftool/dbconfig/20250604-081219-fceratto.json
* 08:11 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2224.codfw.wmnet with reason: Maintenance
* 08:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2224 [[phab:T395989|T395989]]', diff saved to https://phabricator.wikimedia.org/P76990 and previous config saved to /var/cache/conftool/dbconfig/20250604-081058-marostegui.json
* 08:05 moritzm: installing gcc-12 bugfix updates from Bookworm point releases (includes various run time libraries)
* 08:05 marostegui@cumin1002: dbctl commit (dc=all): 'es1035 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76989 and previous config saved to /var/cache/conftool/dbconfig/20250604-080546-root.json
* 08:03 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org
* 08:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org
* 07:57 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76988 and previous config saved to /var/cache/conftool/dbconfig/20250604-075711-fceratto.json
* 07:50 marostegui@cumin1002: dbctl commit (dc=all): 'es1035 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76987 and previous config saved to /var/cache/conftool/dbconfig/20250604-075041-root.json
* 07:48 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76986 and previous config saved to /var/cache/conftool/dbconfig/20250604-074850-fceratto.json
* 07:48 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 07:48 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 07:39 marostegui@cumin1002: dbctl commit (dc=all): 'es2048 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76985 and previous config saved to /var/cache/conftool/dbconfig/20250604-073921-root.json
* 07:37 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of netflow7001.magru.wmnet to plain
* 07:37 jmm@cumin1003: START - Cookbook sre.ganeti.changedisk for changing disk type of netflow7001.magru.wmnet to plain
* 07:35 marostegui@cumin1002: dbctl commit (dc=all): 'es1035 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76984 and previous config saved to /var/cache/conftool/dbconfig/20250604-073535-root.json
* 07:34 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of durum7001.magru.wmnet to plain
* 07:31 jmm@cumin1003: START - Cookbook sre.ganeti.changedisk for changing disk type of durum7001.magru.wmnet to plain
* 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'es2048 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76983 and previous config saved to /var/cache/conftool/dbconfig/20250604-072416-root.json
* 07:23 Emperor: restart swift-object-replicator ms-be2066
* 07:20 marostegui@cumin1002: dbctl commit (dc=all): 'es1035 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76982 and previous config saved to /var/cache/conftool/dbconfig/20250604-072030-root.json
* 07:19 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of doh7001.wikimedia.org to plain
* 07:18 jmm@cumin1003: START - Cookbook sre.ganeti.changedisk for changing disk type of doh7001.wikimedia.org to plain
* 07:16 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ncredir7001.magru.wmnet to plain
* 07:13 jmm@cumin1003: START - Cookbook sre.ganeti.changedisk for changing disk type of ncredir7001.magru.wmnet to plain
* 07:09 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of install7001.wikimedia.org to plain
* 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'es2048 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76981 and previous config saved to /var/cache/conftool/dbconfig/20250604-070910-root.json
* 07:08 jmm@cumin1003: START - Cookbook sre.ganeti.changedisk for changing disk type of install7001.wikimedia.org to plain
* 07:05 marostegui@cumin1002: dbctl commit (dc=all): 'es1035 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76980 and previous config saved to /var/cache/conftool/dbconfig/20250604-070525-root.json
* 07:01 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7001.magru.wmnet
* 06:57 jmm@cumin1003: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7001.magru.wmnet
* 06:54 marostegui@cumin1002: dbctl commit (dc=all): 'es2048 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76979 and previous config saved to /var/cache/conftool/dbconfig/20250604-065405-root.json
* 06:50 marostegui@cumin1002: dbctl commit (dc=all): 'es1035 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76978 and previous config saved to /var/cache/conftool/dbconfig/20250604-065020-root.json
* 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'es2048 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76977 and previous config saved to /var/cache/conftool/dbconfig/20250604-063900-root.json
* 06:35 marostegui@cumin1002: dbctl commit (dc=all): 'es1035 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P76976 and previous config saved to /var/cache/conftool/dbconfig/20250604-063515-root.json
* 06:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 06:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 06:31 marostegui@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153432{{!}}Revert &quot;db-production.php: Disable writes on es7&quot;]] (duration: 09m 52s)
* 06:24 marostegui@deploy1003: marostegui: Continuing with sync
* 06:24 marostegui@deploy1003: marostegui: Backport for [[gerrit:1153432{{!}}Revert &quot;db-production.php: Disable writes on es7&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 06:23 marostegui@cumin1002: dbctl commit (dc=all): 'es2048 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76974 and previous config saved to /var/cache/conftool/dbconfig/20250604-062355-root.json
* 06:21 marostegui@deploy1003: Started scap sync-world: Backport for [[gerrit:1153432{{!}}Revert &quot;db-production.php: Disable writes on es7&quot;]]
* 06:20 marostegui@cumin1002: dbctl commit (dc=all): 'es1035 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P76973 and previous config saved to /var/cache/conftool/dbconfig/20250604-062010-root.json
* 06:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1035 [[phab:T395982|T395982]]', diff saved to https://phabricator.wikimedia.org/P76972 and previous config saved to /var/cache/conftool/dbconfig/20250604-060413-marostegui.json
* 06:03 marostegui@dns1006: END - running authdns-update
* 06:03 marostegui@dns1006: START - running authdns-update
* 06:02 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es1039 to es7 primary [[phab:T395982|T395982]]', diff saved to https://phabricator.wikimedia.org/P76971 and previous config saved to /var/cache/conftool/dbconfig/20250604-060246-marostegui.json
* 06:02 marostegui: Starting es7 eqiad failover from es1035 to es1039 - [[phab:T395982|T395982]]
* 05:57 marostegui@cumin1002: dbctl commit (dc=all): 'Set es1039 with weight 0 [[phab:T395982|T395982]]', diff saved to https://phabricator.wikimedia.org/P76970 and previous config saved to /var/cache/conftool/dbconfig/20250604-055744-marostegui.json
* 05:56 marostegui@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153431{{!}}db-production.php: Disable writes on es7 (T395982)]] (duration: 13m 00s)
* 05:49 marostegui@deploy1003: marostegui: Continuing with sync
* 05:45 marostegui@deploy1003: marostegui: Backport for [[gerrit:1153431{{!}}db-production.php: Disable writes on es7 (T395982)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 05:43 marostegui@deploy1003: Started scap sync-world: Backport for [[gerrit:1153431{{!}}db-production.php: Disable writes on es7 (T395982)]]
* 05:42 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es7 [[phab:T395982|T395982]]
* 00:38 eileen: civicrm upgraded from {{Gerrit|8eb67a94}} to {{Gerrit|22171c0b}}

== 2025-06-03 ==
* 22:42 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:39 andrew@cumin1002: START - Cookbook sre.dns.netbox
* 22:10 eileen: civicrm upgraded from {{Gerrit|3b59e784}} to {{Gerrit|8eb67a94}}
* 22:09 sbassett@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 21:59 sbassett@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 21:53 tzatziki: removing 4 files for legal compliance
* 21:48 sbassett@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 21:41 tzatziki: removing 2 files for legal compliance
* 21:38 sbassett@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 21:38 sbassett@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 21:28 sbassett@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 21:21 mstyles@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153351{{!}}Revert^2 &quot;OATHAuth: Mark checkuser and suppress as requiring 2FA&quot; (T150898)]] (duration: 11m 31s)
* 21:18 bvibber@deploy1003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 21:14 mstyles@deploy1003: mstyles, sbassett: Continuing with sync
* 21:11 mstyles@deploy1003: mstyles, sbassett: Backport for [[gerrit:1153351{{!}}Revert^2 &quot;OATHAuth: Mark checkuser and suppress as requiring 2FA&quot; (T150898)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:09 mstyles@deploy1003: Started scap sync-world: Backport for [[gerrit:1153351{{!}}Revert^2 &quot;OATHAuth: Mark checkuser and suppress as requiring 2FA&quot; (T150898)]]
* 21:03 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153350{{!}}Use default preference if no client preference in auth request (T395957)]] (duration: 09m 49s)
* 20:56 cjming@deploy1003: matmarex, cjming: Continuing with sync
* 20:55 cjming@deploy1003: matmarex, cjming: Backport for [[gerrit:1153350{{!}}Use default preference if no client preference in auth request (T395957)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:53 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1153350{{!}}Use default preference if no client preference in auth request (T395957)]]
* 20:46 marostegui@cumin1002: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of es2040.codfw.wmnet onto es2048.codfw.wmnet
* 20:46 marostegui@cumin1002: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) es2048 gradually with 4 steps - Pool es2048.codfw.wmnet in after cloning
* 20:37 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153341{{!}}Use ::getContentId() and ::clearContentId() from the Parsoid extension API]] (duration: 12m 41s)
* 20:31 ryankemper@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_main on wdqs1022.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/rdf_subgraphs/snapshot=20250526/wiki=wikidata/scope=wikidata_main/ using stat1009.eqiad.wmnet)
* 20:30 cscott@deploy1003: cscott: Continuing with sync
* 20:27 cscott@deploy1003: cscott: Backport for [[gerrit:1153341{{!}}Use ::getContentId() and ::clearContentId() from the Parsoid extension API]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:25 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1153341{{!}}Use ::getContentId() and ::clearContentId() from the Parsoid extension API]]
* 20:18 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152860{{!}}Deploy survey to en at twenty percent (T389393)]] (duration: 11m 18s)
* 20:11 cjming@deploy1003: ksarabia, cjming: Continuing with sync
* 20:09 ryankemper@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading scholarly_articles on wdqs1023.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/rdf_subgraphs/snapshot=20250526/wiki=wikidata/scope=scholarly_articles/ using stat1011.eqiad.wmnet)
* 20:08 cjming@deploy1003: ksarabia, cjming: Backport for [[gerrit:1152860{{!}}Deploy survey to en at twenty percent (T389393)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:06 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1152860{{!}}Deploy survey to en at twenty percent (T389393)]]
* 19:37 ryankemper@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs1020.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/rdf/date=20250526/wiki=wikidata/ using stat1008.eqiad.wmnet)
* 19:37 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs1020.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/rdf/date=20250526/wiki=wikidata/ using stat1008.eqiad.wmnet)
* 19:36 ryankemper@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata_full on wdqs1020.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/rdf/date=20250526/wiki=wikidata/ using stat1010.eqiad.wmnet)
* 19:36 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs1020.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/rdf/date=20250526/wiki=wikidata/ using stat1010.eqiad.wmnet)
* 19:35 ryankemper@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading wikidata on wdqs1020.eqiad.wmnet from DumpsSource.NFS (munging data to /srv/wdqs/munged, /srv/wdqs/lex-munged)
* 19:35 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-reload reloading wikidata on wdqs1020.eqiad.wmnet from DumpsSource.NFS (munging data to /srv/wdqs/munged, /srv/wdqs/lex-munged)
* 19:22 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.4  refs [[phab:T392174|T392174]]
* 19:11 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-reload reloading scholarly_articles on wdqs1023.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/rdf_subgraphs/snapshot=20250526/wiki=wikidata/scope=scholarly_articles/ using stat1011.eqiad.wmnet)
* 19:08 ryankemper@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) reloading scholarly_articles on wdqs1023.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/rdf_subgraphs/snapshot=20250526/wiki=wikidata/scope=scholarly_articles/ using stat1009.eqiad.wmnet)
* 19:07 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-reload reloading scholarly_articles on wdqs1023.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/rdf_subgraphs/snapshot=20250526/wiki=wikidata/scope=scholarly_articles/ using stat1009.eqiad.wmnet)
* 19:05 ebernhardson@deploy1003: Finished deploy [wdqs/wdqs@fea7794]: 0.3.157 (duration: 17m 57s)
* 19:00 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-reload reloading wikidata_main on wdqs1022.eqiad.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/rdf_subgraphs/snapshot=20250526/wiki=wikidata/scope=wikidata_main/ using stat1009.eqiad.wmnet)
* 18:47 ebernhardson@deploy1003: Started deploy [wdqs/wdqs@fea7794]: 0.3.157
* 18:08 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet
* 18:00 aokoth@cumin1002: START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet
* 17:50 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 17:50 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 17:45 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 17:45 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 17:42 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 17:41 isaranto@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 17:39 bvibber@deploy1003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 17:39 bvibber@deploy1003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 17:39 isaranto@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
* 17:39 bvibber@deploy1003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 17:38 bvibber@deploy1003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 17:38 bvibber@deploy1003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 17:36 swfrench@deploy1003: Finished scap sync-world: Scap test run after revert - [[phab:T389786|T389786]] (duration: 02m 10s)
* 17:35 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1019.eqiad.wmnet,service=s4
* 17:34 swfrench@deploy1003: Started scap sync-world: Scap test run after revert - [[phab:T389786|T389786]]
* 17:19 swfrench@deploy1003: Started scap sync-world: Scap run to test newly enabled dse-k8s-eqiad deployment - [[phab:T388761|T388761]] [[phab:T389786|T389786]]
* 17:17 bvibber@deploy1003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 17:16 bvibber@deploy1003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 17:16 bvibber@deploy1003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 17:15 cdanis@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:15 cdanis@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:14 bvibber@deploy1003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 17:14 cdanis@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:13 bvibber@deploy1003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 17:13 bvibber@deploy1003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 17:12 cdanis@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:06 cdanis@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:06 cdanis@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:01 cdanis@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 16:59 cdanis@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 16:57 fnegri@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1019.eqiad.wmnet with reason: Debugging stuck queryies [[phab:T390767|T390767]]
* 16:42 bvibber@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153282{{!}}Fixes: Charts embedded in template rendering in Parsoid (T395462)]], [[gerrit:1153281{{!}}Fixes: Charts embedded in template rendering in Parsoid (T395462)]] (duration: 09m 54s)
* 16:38 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcontrol2010-dev.codfw.wmnet with OS bullseye
* 16:35 bvibber@deploy1003: bvibber: Continuing with sync
* 16:35 bvibber@deploy1003: bvibber: Backport for [[gerrit:1153282{{!}}Fixes: Charts embedded in template rendering in Parsoid (T395462)]], [[gerrit:1153281{{!}}Fixes: Charts embedded in template rendering in Parsoid (T395462)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:34 sukhe@dns1004: END - running authdns-update
* 16:33 sukhe: testing dummy authdns-update to ensure clean run after gc-authdns-git-repo.timer rnu
* 16:33 sukhe@dns1004: START - running authdns-update
* 16:32 bvibber@deploy1003: Started scap sync-world: Backport for [[gerrit:1153282{{!}}Fixes: Charts embedded in template rendering in Parsoid (T395462)]], [[gerrit:1153281{{!}}Fixes: Charts embedded in template rendering in Parsoid (T395462)]]
* 16:23 jiji@deploy1003: Finished scap sync-world: [[phab:T276994|T276994]]: We merged a number of noop patches, sparing deployers the scary diffs (duration: 02m 58s)
* 16:22 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 16:21 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 16:20 jiji@deploy1003: Started scap sync-world: [[phab:T276994|T276994]]: We merged a number of noop patches, sparing deployers the scary diffs
* 16:12 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
* 16:12 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
* 16:06 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 16:04 jiji@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 16:04 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 15:57 jiji@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 15:57 jiji@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 15:55 jiji@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 15:54 jiji@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 15:51 jiji@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 15:51 jiji@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 15:50 jiji@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 15:18 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcontrol2010-dev.codfw.wmnet with OS bullseye
* 15:18 moritzm: installing gcc-12 bugfix updates from Bookworm point releases (includes various run time libraries)
* 15:17 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcontrol2010-dev.codfw.wmnet with OS bullseye
* 15:16 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 15:15 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76966 and previous config saved to /var/cache/conftool/dbconfig/20250603-151552-fceratto.json
* 15:10 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;add prometheus7002 - jmm@cumin1003&quot;
* 15:10 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;add prometheus7002 - jmm@cumin1003&quot;
* 15:10 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcontrol2010-dev.codfw.wmnet with OS bullseye
* 15:06 hashar: Restarted Gerrit due to issue with replication config {{!}} [[phab:T395887|T395887]]
* 15:00 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P76964 and previous config saved to /var/cache/conftool/dbconfig/20250603-150045-fceratto.json
* 14:58 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;add prometheus7002 - jmm@cumin1003&quot;
* 14:58 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;add prometheus7002 - jmm@cumin1003&quot;
* 14:54 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;add prometheus7002 - jmm@cumin1003&quot;
* 14:54 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;add prometheus7002 - jmm@cumin1003&quot;
* 14:50 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet,service=s4
* 14:46 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host prometheus7002.magru.wmnet with OS bookworm
* 14:45 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P76963 and previous config saved to /var/cache/conftool/dbconfig/20250603-144538-fceratto.json
* 14:30 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76962 and previous config saved to /var/cache/conftool/dbconfig/20250603-143031-fceratto.json
* 14:27 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on prometheus7002.magru.wmnet with reason: host reimage
* 14:26 jmm@cumin1003: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: elastic1063.eqiad.wmnet
* 14:23 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on prometheus7002.magru.wmnet with reason: host reimage
* 14:23 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76961 and previous config saved to /var/cache/conftool/dbconfig/20250603-142314-fceratto.json
* 14:23 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 14:22 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76960 and previous config saved to /var/cache/conftool/dbconfig/20250603-142248-fceratto.json
* 14:19 jmm@cumin1003: DONE (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: elastic1103.eqiad.wmnet
* 14:07 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P76959 and previous config saved to /var/cache/conftool/dbconfig/20250603-140740-fceratto.json
* 14:01 Amir1: dropping term store tables from s8 ([[phab:T351820|T351820]])
* 14:01 Amir1: dropping term store tables from s8 ([[phab:T351802|T351802]])
* 13:57 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host prometheus7002.magru.wmnet with OS bookworm
* 13:52 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P76957 and previous config saved to /var/cache/conftool/dbconfig/20250603-135233-fceratto.json
* 13:49 marostegui@cumin1002: dbctl commit (dc=all): 'es1039 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76956 and previous config saved to /var/cache/conftool/dbconfig/20250603-134935-root.json
* 13:44 phuedx: Disabled the SDS 2.4.11 Synthetic A/A Test in xLab
* 13:42 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 13:40 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1048.eqiad.wmnet with OS bullseye
* 13:38 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 13:37 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76954 and previous config saved to /var/cache/conftool/dbconfig/20250603-133725-fceratto.json
* 13:34 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 13:34 marostegui@cumin1002: dbctl commit (dc=all): 'es1039 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76953 and previous config saved to /var/cache/conftool/dbconfig/20250603-133429-root.json
* 13:33 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 13:32 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 13:32 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 13:31 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 13:28 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76952 and previous config saved to /var/cache/conftool/dbconfig/20250603-132802-fceratto.json
* 13:27 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 13:27 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 13:27 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76951 and previous config saved to /var/cache/conftool/dbconfig/20250603-132735-fceratto.json
* 13:26 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 13:22 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 13:21 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 13:20 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
* 13:20 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 13:20 marostegui@cumin1002: dbctl commit (dc=all): 'es2038 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76950 and previous config saved to /var/cache/conftool/dbconfig/20250603-131959-root.json
* 13:19 marostegui@cumin1002: dbctl commit (dc=all): 'es1039 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76949 and previous config saved to /var/cache/conftool/dbconfig/20250603-131923-root.json
* 13:18 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 13:17 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 13:16 moritzm: installing libavif security updates
* 13:16 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1048.eqiad.wmnet with reason: host reimage
* 13:15 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 13:14 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
* 13:14 jgleeson: payments-wiki rolled back from {{Gerrit|def6c267}} to {{Gerrit|1a4ef678}}
* 13:12 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 13:12 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1048.eqiad.wmnet with reason: host reimage
* 13:12 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 13:12 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P76948 and previous config saved to /var/cache/conftool/dbconfig/20250603-131228-fceratto.json
* 13:11 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 13:11 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 13:08 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 13:04 marostegui@cumin1002: dbctl commit (dc=all): 'es2038 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76946 and previous config saved to /var/cache/conftool/dbconfig/20250603-130453-root.json
* 13:04 marostegui@cumin1002: dbctl commit (dc=all): 'es1039 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76945 and previous config saved to /var/cache/conftool/dbconfig/20250603-130418-root.json
* 13:04 marostegui: Shutdown clouddb1016:x3 [[phab:T390954|T390954]]
* 13:02 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1016.eqiad.wmnet with reason: Setting up x3 [[phab:T390954|T390954]]
* 12:58 moritzm: uploaded wmf-laptop 1.0.2 to apt.wikimedia.org
* 12:57 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P76943 and previous config saved to /var/cache/conftool/dbconfig/20250603-125721-fceratto.json
* 12:49 marostegui@cumin1002: dbctl commit (dc=all): 'es2038 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76942 and previous config saved to /var/cache/conftool/dbconfig/20250603-124948-root.json
* 12:49 marostegui@cumin1002: dbctl commit (dc=all): 'es1039 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76941 and previous config saved to /var/cache/conftool/dbconfig/20250603-124913-root.json
* 12:42 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76940 and previous config saved to /var/cache/conftool/dbconfig/20250603-124214-fceratto.json
* 12:40 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2049.codfw.wmnet
* 12:35 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2049.codfw.wmnet
* 12:34 marostegui@cumin1002: dbctl commit (dc=all): 'es2038 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76939 and previous config saved to /var/cache/conftool/dbconfig/20250603-123442-root.json
* 12:34 marostegui@cumin1002: dbctl commit (dc=all): 'es1039 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76938 and previous config saved to /var/cache/conftool/dbconfig/20250603-123407-root.json
* 12:33 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76937 and previous config saved to /var/cache/conftool/dbconfig/20250603-123357-fceratto.json
* 12:33 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 12:33 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76936 and previous config saved to /var/cache/conftool/dbconfig/20250603-123331-fceratto.json
* 12:30 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1048.eqiad.wmnet with OS bullseye
* 12:25 marostegui@cumin1002: START - Cookbook sre.mysql.pool es2048 gradually with 4 steps - Pool es2048.codfw.wmnet in after cloning
* 12:19 marostegui@cumin1002: dbctl commit (dc=all): 'es2038 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76935 and previous config saved to /var/cache/conftool/dbconfig/20250603-121937-root.json
* 12:19 marostegui@cumin1002: dbctl commit (dc=all): 'es1039 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P76934 and previous config saved to /var/cache/conftool/dbconfig/20250603-121902-root.json
* 12:18 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P76933 and previous config saved to /var/cache/conftool/dbconfig/20250603-121824-fceratto.json
* 12:16 marostegui@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153135{{!}}Revert &quot;db-production.php: Disable writes on es7&quot;]] (duration: 09m 47s)
* 12:15 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
* 12:12 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' .
* 12:10 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' .
* 12:09 marostegui@deploy1003: marostegui: Continuing with sync
* 12:09 marostegui@deploy1003: marostegui: Backport for [[gerrit:1153135{{!}}Revert &quot;db-production.php: Disable writes on es7&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:08 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'readability' for release 'main' .
* 12:07 claime: Launching manual run of recount-categories cronjob - [[phab:T395745|T395745]]
* 12:06 marostegui@deploy1003: Started scap sync-world: Backport for [[gerrit:1153135{{!}}Revert &quot;db-production.php: Disable writes on es7&quot;]]
* 12:06 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 12:05 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 12:04 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' .
* 12:04 marostegui@cumin1002: dbctl commit (dc=all): 'es2038 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76931 and previous config saved to /var/cache/conftool/dbconfig/20250603-120431-root.json
* 12:03 marostegui@cumin1002: dbctl commit (dc=all): 'es1039 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P76930 and previous config saved to /var/cache/conftool/dbconfig/20250603-120356-root.json
* 12:03 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P76929 and previous config saved to /var/cache/conftool/dbconfig/20250603-120316-fceratto.json
* 12:02 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' .
* 12:00 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' .
* 11:59 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 11:56 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' .
* 11:55 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' .
* 11:54 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
* 11:52 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
* 11:50 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2038 [[phab:T395785|T395785]]', diff saved to https://phabricator.wikimedia.org/P76927 and previous config saved to /var/cache/conftool/dbconfig/20250603-115026-marostegui.json
* 11:49 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
* 11:49 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es2039 to es7 primary and set section read-write [[phab:T395785|T395785]]', diff saved to https://phabricator.wikimedia.org/P76926 and previous config saved to /var/cache/conftool/dbconfig/20250603-114917-marostegui.json
* 11:48 marostegui: Starting es7 codfw failover from es2038 to es2039 - [[phab:T395785|T395785]]
* 11:48 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76925 and previous config saved to /var/cache/conftool/dbconfig/20250603-114809-fceratto.json
* 11:46 marostegui@cumin1002: dbctl commit (dc=all): 'Set es2039 with weight 0 [[phab:T395785|T395785]]', diff saved to https://phabricator.wikimedia.org/P76924 and previous config saved to /var/cache/conftool/dbconfig/20250603-114637-marostegui.json
* 11:46 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es7 [[phab:T395785|T395785]]
* 11:46 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' .
* 11:41 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2048.codfw.wmnet
* 11:40 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
* 11:39 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76923 and previous config saved to /var/cache/conftool/dbconfig/20250603-113952-fceratto.json
* 11:39 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1039 [[phab:T395647|T395647]]', diff saved to https://phabricator.wikimedia.org/P76922 and previous config saved to /var/cache/conftool/dbconfig/20250603-113946-marostegui.json
* 11:39 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 11:39 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76921 and previous config saved to /var/cache/conftool/dbconfig/20250603-113924-fceratto.json
* 11:39 marostegui@deploy1003: Finished scap sync-world: Backport for [[gerrit:1153130{{!}}db-production.php: Disable writes on es7 (T395647)]] (duration: 09m 56s)
* 11:36 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2048.codfw.wmnet
* 11:33 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
* 11:32 marostegui@deploy1003: marostegui: Continuing with sync
* 11:31 marostegui@deploy1003: marostegui: Backport for [[gerrit:1153130{{!}}db-production.php: Disable writes on es7 (T395647)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:29 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es7 [[phab:T395785|T395785]]
* 11:29 marostegui@deploy1003: Started scap sync-world: Backport for [[gerrit:1153130{{!}}db-production.php: Disable writes on es7 (T395647)]]
* 11:25 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) es2040 gradually with 4 steps - Pool es2040.codfw.wmnet in after cloning
* 11:22 taavi@cumin1002: conftool action : set/weight=100:pooled=yes; selector: name=clouddb1016.eqiad.wmnet,service=x3
* 11:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es7 [[phab:T395785|T395785]]
* 11:18 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P76919 and previous config saved to /var/cache/conftool/dbconfig/20250603-111822-fceratto.json
* 11:09 fnegri@cumin1002: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
* 11:03 jgleeson: payments-wiki upgraded from {{Gerrit|1a4ef678}} to {{Gerrit|def6c267}}
* 11:03 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P76917 and previous config saved to /var/cache/conftool/dbconfig/20250603-110315-fceratto.json
* 10:57 phuedx: Enabling the SDS 2.4.11 Synthetic A/A Test in xLab
* 10:57 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2047.codfw.wmnet
* 10:52 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2047.codfw.wmnet
* 10:49 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2046.codfw.wmnet
* 10:48 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76915 and previous config saved to /var/cache/conftool/dbconfig/20250603-104809-fceratto.json
* 10:44 fnegri@cumin1002: START - Cookbook sre.wikireplicas.update-views
* 10:44 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2046.codfw.wmnet
* 10:43 fnegri@cumin1002: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
* 10:43 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2045.codfw.wmnet
* 10:40 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76914 and previous config saved to /var/cache/conftool/dbconfig/20250603-104056-fceratto.json
* 10:40 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 10:40 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76913 and previous config saved to /var/cache/conftool/dbconfig/20250603-104030-fceratto.json
* 10:40 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti2045.codfw.wmnet
* 10:38 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 11 hosts with reason: silence alerts due to down BGP groups on cr2-codfw while PIC is reconfigured
* 10:25 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P76912 and previous config saved to /var/cache/conftool/dbconfig/20250603-102523-fceratto.json
* 10:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Fix the repool dbctl commit', diff saved to https://phabricator.wikimedia.org/P76911 and previous config saved to /var/cache/conftool/dbconfig/20250603-102517-ladsgroup.json
* 10:18 fnegri@cumin1002: START - Cookbook sre.wikireplicas.update-views
* 10:13 topranks: drain cr2-codfw traffic to enable PIC port bw rebalence on slot 0 [[phab:T387504|T387504]]
* 09:59 jmm@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host prometheus7002.magru.wmnet with OS bookworm
* 09:43 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner1002.eqiad.wmnet with OS bookworm
* 09:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner1002.eqiad.wmnet with reason: host reimage
* 09:22 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner1002.eqiad.wmnet with reason: host reimage
* 09:22 elukey: puppet cert destroy &lt;nowiki&gt;{&lt;/nowiki&gt;mobileapps,proton,recommendation-api&lt;nowiki&gt;}&lt;/nowiki&gt;.discovery.wmnet on puppetmaster1001 - old certs not used anymore
* 09:18 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:18 marostegui@cumin1002: START - Cookbook sre.mysql.pool es2040 gradually with 4 steps - Pool es2040.codfw.wmnet in after cloning
* 09:18 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:16 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:16 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:15 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P76909 and previous config saved to /var/cache/conftool/dbconfig/20250603-091521-fceratto.json
* 09:10 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:10 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 09:06 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be2008.codfw.wmnet with OS bullseye
* 09:06 mvernon@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1002&quot;
* 09:00 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76907 and previous config saved to /var/cache/conftool/dbconfig/20250603-090013-fceratto.json
* 08:59 mvernon@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1002&quot;
* 08:51 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76906 and previous config saved to /var/cache/conftool/dbconfig/20250603-085148-fceratto.json
* 08:51 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 08:51 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76905 and previous config saved to /var/cache/conftool/dbconfig/20250603-085121-fceratto.json
* 08:45 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:43 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:42 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be2008.codfw.wmnet with reason: host reimage
* 08:41 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:40 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 08:38 mvernon@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be2008.codfw.wmnet with reason: host reimage
* 08:37 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 08:36 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P76904 and previous config saved to /var/cache/conftool/dbconfig/20250603-083614-fceratto.json
* 08:36 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 08:34 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host prometheus7002.magru.wmnet with OS bookworm
* 08:30 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host gitlab-runner1002.eqiad.wmnet with OS bookworm
* 08:22 moritzm: rearm keyholder on cumin1003 following reboot
* 08:21 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P76903 and previous config saved to /var/cache/conftool/dbconfig/20250603-082107-fceratto.json
* 08:16 mvernon@cumin1002: START - Cookbook sre.hosts.reimage for host thanos-be2008.codfw.wmnet with OS bullseye
* 08:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin1003.eqiad.wmnet
* 08:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cumin1003.eqiad.wmnet
* 08:10 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be2009.codfw.wmnet with OS bullseye
* 08:10 mvernon@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1002&quot;
* 08:06 mvernon@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - mvernon@cumin1002&quot;
* 08:06 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76901 and previous config saved to /var/cache/conftool/dbconfig/20250603-080600-fceratto.json
* 08:04 phuedx: Disabling the SDS 2.4.11 Synthetic A/A/ Test in xLab
* 07:57 phuedx: Enabling the SDS 2.4.11 Synthetic A/A Test in xLab
* 07:56 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76900 and previous config saved to /var/cache/conftool/dbconfig/20250603-075638-fceratto.json
* 07:56 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 07:56 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host prometheus7002.magru.wmnet
* 07:56 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76899 and previous config saved to /var/cache/conftool/dbconfig/20250603-075622-fceratto.json
* 07:56 jmm@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 07:49 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be2009.codfw.wmnet with reason: host reimage
* 07:49 jmm@cumin1003: END (PASS) - Cookbook sre.misc-clusters.restart-reboot-config-master (exit_code=0) rolling reboot on A:config-master
* 07:46 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 07:46 jmm@cumin1003: START - Cookbook sre.ganeti.makevm for new host prometheus7002.magru.wmnet
* 07:46 mvernon@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be2009.codfw.wmnet with reason: host reimage
* 07:44 jmm@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors
* 07:44 jmm@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors
* 07:43 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host bast7002.wikimedia.org
* 07:43 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host bast7002.wikimedia.org with OS bookworm
* 07:41 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P76898 and previous config saved to /var/cache/conftool/dbconfig/20250603-074113-fceratto.json
* 07:38 jmm@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) config-master.discovery.wmnet. on all recursors
* 07:38 jmm@cumin1003: START - Cookbook sre.dns.wipe-cache config-master.discovery.wmnet. on all recursors
* 07:37 jmm@cumin1003: START - Cookbook sre.misc-clusters.restart-reboot-config-master rolling reboot on A:config-master
* 07:27 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast7002.wikimedia.org with reason: host reimage
* 07:26 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P76897 and previous config saved to /var/cache/conftool/dbconfig/20250603-072604-fceratto.json
* 07:25 tchanders@deploy1003: Finished scap sync-world: Backport for [[gerrit:1142649{{!}}Assign IP auto-reveal rights to certain groups (T386492)]] (duration: 10m 39s)
* 07:23 mvernon@cumin1002: START - Cookbook sre.hosts.reimage for host thanos-be2009.codfw.wmnet with OS bullseye
* 07:23 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on bast7002.wikimedia.org with reason: host reimage
* 07:18 tchanders@deploy1003: tchanders: Continuing with sync
* 07:16 tchanders@deploy1003: tchanders: Backport for [[gerrit:1142649{{!}}Assign IP auto-reveal rights to certain groups (T386492)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:14 tchanders@deploy1003: Started scap sync-world: Backport for [[gerrit:1142649{{!}}Assign IP auto-reveal rights to certain groups (T386492)]]
* 07:10 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76896 and previous config saved to /var/cache/conftool/dbconfig/20250603-071057-fceratto.json
* 07:06 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:06 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:02 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:02 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 07:01 marostegui@cumin1002: dbctl commit (dc=all): 'es1037 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76895 and previous config saved to /var/cache/conftool/dbconfig/20250603-070155-root.json
* 07:00 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76894 and previous config saved to /var/cache/conftool/dbconfig/20250603-070036-fceratto.json
* 07:00 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 07:00 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76893 and previous config saved to /var/cache/conftool/dbconfig/20250603-070021-fceratto.json
* 06:58 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host bast7002.wikimedia.org with OS bookworm
* 06:54 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM bast7002.wikimedia.org - jmm@cumin1003&quot;
* 06:54 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM bast7002.wikimedia.org - jmm@cumin1003&quot;
* 06:54 jmm@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) bast7002.wikimedia.org on all recursors
* 06:54 jmm@cumin1003: START - Cookbook sre.dns.wipe-cache bast7002.wikimedia.org on all recursors
* 06:54 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 06:54 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM bast7002.wikimedia.org - jmm@cumin1003&quot;
* 06:53 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM bast7002.wikimedia.org - jmm@cumin1003&quot;
* 06:48 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-be2009.codfw.wmnet with OS bullseye
* 06:46 marostegui@cumin1002: dbctl commit (dc=all): 'es1037 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76892 and previous config saved to /var/cache/conftool/dbconfig/20250603-064649-root.json
* 06:45 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 06:45 jmm@cumin1003: START - Cookbook sre.ganeti.makevm for new host bast7002.wikimedia.org
* 06:45 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P76891 and previous config saved to /var/cache/conftool/dbconfig/20250603-064513-fceratto.json
* 06:41 marostegui@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76890 and previous config saved to /var/cache/conftool/dbconfig/20250603-064147-root.json
* 06:37 marostegui: Decrease buffer size on clouddb1016:s8 [[phab:T390954|T390954]]
* 06:31 marostegui@cumin1002: dbctl commit (dc=all): 'es1037 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76889 and previous config saved to /var/cache/conftool/dbconfig/20250603-063144-root.json
* 06:30 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1154.eqiad.wmnet with reason: Setting up x3 [[phab:T390954|T390954]]
* 06:30 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P76887 and previous config saved to /var/cache/conftool/dbconfig/20250603-063004-fceratto.json
* 06:29 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1016.eqiad.wmnet with reason: Setting up x3 [[phab:T390954|T390954]]
* 06:26 marostegui@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76886 and previous config saved to /var/cache/conftool/dbconfig/20250603-062641-root.json
* 06:16 marostegui@cumin1002: dbctl commit (dc=all): 'es1037 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76885 and previous config saved to /var/cache/conftool/dbconfig/20250603-061638-root.json
* 06:14 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76884 and previous config saved to /var/cache/conftool/dbconfig/20250603-061457-fceratto.json
* 06:11 marostegui@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76883 and previous config saved to /var/cache/conftool/dbconfig/20250603-061134-root.json
* 06:07 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db2152 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76882 and previous config saved to /var/cache/conftool/dbconfig/20250603-060719-fceratto.json
* 06:07 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance
* 06:01 marostegui@cumin1002: dbctl commit (dc=all): 'es1037 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76881 and previous config saved to /var/cache/conftool/dbconfig/20250603-060132-root.json
* 05:56 marostegui@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76880 and previous config saved to /var/cache/conftool/dbconfig/20250603-055628-root.json
* 05:46 marostegui@cumin1002: dbctl commit (dc=all): 'es1037 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76879 and previous config saved to /var/cache/conftool/dbconfig/20250603-054626-root.json
* 05:41 marostegui@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76878 and previous config saved to /var/cache/conftool/dbconfig/20250603-054123-root.json
* 05:39 marostegui@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152896{{!}}Revert &quot;db-production.php: Disable writes on es6&quot;]] (duration: 09m 52s)
* 05:32 marostegui@deploy1003: marostegui: Continuing with sync
* 05:31 marostegui@cumin1002: dbctl commit (dc=all): 'Give some weight to es1038', diff saved to https://phabricator.wikimedia.org/P76877 and previous config saved to /var/cache/conftool/dbconfig/20250603-053151-marostegui.json
* 05:31 marostegui@deploy1003: marostegui: Backport for [[gerrit:1152896{{!}}Revert &quot;db-production.php: Disable writes on es6&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 05:29 marostegui@deploy1003: Started scap sync-world: Backport for [[gerrit:1152896{{!}}Revert &quot;db-production.php: Disable writes on es6&quot;]]
* 05:27 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1037 [[phab:T395867|T395867]]', diff saved to https://phabricator.wikimedia.org/P76876 and previous config saved to /var/cache/conftool/dbconfig/20250603-052719-marostegui.json
* 05:27 marostegui@dns1006: END - running authdns-update
* 05:26 marostegui@dns1006: START - running authdns-update
* 05:26 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es1038 to es6 primary [[phab:T395867|T395867]]', diff saved to https://phabricator.wikimedia.org/P76875 and previous config saved to /var/cache/conftool/dbconfig/20250603-052614-marostegui.json
* 05:25 marostegui: Starting es6 eqiad failover from es1037 to es1038 - [[phab:T395867|T395867]]
* 05:23 marostegui@cumin1002: dbctl commit (dc=all): 'Set es1038 with weight 0 [[phab:T395867|T395867]]', diff saved to https://phabricator.wikimedia.org/P76874 and previous config saved to /var/cache/conftool/dbconfig/20250603-052353-marostegui.json
* 05:22 marostegui@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152893{{!}}db-production.php: Disable writes on es6 (T395867)]] (duration: 13m 39s)
* 05:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es6 [[phab:T395867|T395867]]
* 05:14 marostegui@deploy1003: marostegui: Continuing with sync
* 05:13 marostegui@deploy1003: marostegui: Backport for [[gerrit:1152893{{!}}db-production.php: Disable writes on es6 (T395867)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 05:09 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es6 [[phab:T395867|T395867]]
* 05:09 marostegui@deploy1003: Started scap sync-world: Backport for [[gerrit:1152893{{!}}db-production.php: Disable writes on es6 (T395867)]]
* 04:54 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es2037.codfw.wmnet with reason: Primary switchover es6 [[phab:T395420|T395420]]
* 04:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2037 [[phab:T395420|T395420]]', diff saved to https://phabricator.wikimedia.org/P76873 and previous config saved to /var/cache/conftool/dbconfig/20250603-045251-marostegui.json
* 04:52 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es2035 to es6 primary and set section read-write [[phab:T395420|T395420]]', diff saved to https://phabricator.wikimedia.org/P76872 and previous config saved to /var/cache/conftool/dbconfig/20250603-045202-root.json
* 04:51 marostegui: Starting es6 codfw failover from es2037 to es2035 - [[phab:T395420|T395420]]
* 04:49 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es6 [[phab:T395420|T395420]]
* 04:48 marostegui@cumin1002: dbctl commit (dc=all): 'Set es2035 with weight 0 [[phab:T395420|T395420]]', diff saved to https://phabricator.wikimedia.org/P76871 and previous config saved to /var/cache/conftool/dbconfig/20250603-044855-root.json
* 04:45 marostegui@cumin1002: dbctl commit (dc=all): 'Add es2048 to dbctl depooled [[phab:T395771|T395771]]+', diff saved to https://phabricator.wikimedia.org/P76870 and previous config saved to /var/cache/conftool/dbconfig/20250603-044550-marostegui.json
* 04:40 marostegui@cumin1002: START - Cookbook sre.mysql.clone of es2040.codfw.wmnet onto es2048.codfw.wmnet
* 04:31 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2040 [[phab:T395771|T395771]]', diff saved to https://phabricator.wikimedia.org/P76869 and previous config saved to /var/cache/conftool/dbconfig/20250603-043151-marostegui.json
* 04:29 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2040.codfw.wmnet with reason: Maintenance
* 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.45.0-wmf.1 (duration: 01m 39s)
* 03:48 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.45.0-wmf.4  refs [[phab:T392174|T392174]] (duration: 45m 55s)
* 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.45.0-wmf.4  refs [[phab:T392174|T392174]]
* 00:47 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1048.eqiad.wmnet with reason: host reimage
* 00:45 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1048.eqiad.wmnet with reason: host reimage

== 2025-06-02 ==
* 23:53 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be2008.codfw.wmnet with reason: host reimage
* 23:50 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be2008.codfw.wmnet with reason: host reimage
* 23:30 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host thanos-be2008.codfw.wmnet with OS bullseye
* 23:25 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1048.eqiad.wmnet with OS bullseye
* 23:24 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1051.eqiad.wmnet with OS bullseye
* 23:24 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002&quot;
* 23:24 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002&quot;
* 23:22 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1050.eqiad.wmnet with OS bullseye
* 23:22 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002&quot;
* 23:22 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002&quot;
* 23:10 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1049.eqiad.wmnet with OS bullseye
* 23:10 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002&quot;
* 23:09 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002&quot;
* 22:54 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1051.eqiad.wmnet with reason: host reimage
* 22:50 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1051.eqiad.wmnet with reason: host reimage
* 22:47 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1050.eqiad.wmnet with reason: host reimage
* 22:45 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1050.eqiad.wmnet with reason: host reimage
* 22:38 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1049.eqiad.wmnet with reason: host reimage
* 22:35 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1049.eqiad.wmnet with reason: host reimage
* 22:29 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1051.eqiad.wmnet with OS bullseye
* 22:16 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1048.eqiad.wmnet with OS bullseye
* 22:08 maryum: scap sync-world finished to deploy several security bugs and PrivateSettings.php changes
* 22:01 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1050.eqiad.wmnet with OS bullseye
* 21:53 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1049.eqiad.wmnet with OS bullseye
* 21:38 tgr@deploy1003: Unlocked for deployment [MediaWiki]: [[phab:T395758|T395758]] (duration: 22m 32s)
* 21:38 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1048.eqiad.wmnet with reason: host reimage
* 21:34 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1048.eqiad.wmnet with reason: host reimage
* 21:22 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cirrussearch205*,cirrussearch2060* for [[phab:T395855|T395855]] - bking@cumin2002
* 21:22 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Banning hosts: cirrussearch205*,cirrussearch2060* for [[phab:T395855|T395855]] - bking@cumin2002
* 21:16 bking@cumin2002: conftool action : set/pooled=no; selector: name=cirrussearch2055.codfw.wmnet{{!}}cirrussearch2056.codfw.wmnet{{!}}cirrussearch2057.codfw.wmnet{{!}}cirrussearch2058.codfw.wmnet{{!}}cirrussearch2059.codfw.wmnet{{!}}cirrussearch2060.codfw.wmnet{{!}}cirrussearch2091.codfw.wmnet
* 21:16 tgr@deploy1003: Locking from deployment [MediaWiki]: [[phab:T395758|T395758]]
* 21:13 bking@cumin2002: conftool action : set/pooled=yes:weight=10; selector: name=cirrussearch.*.codfw.wmnet
* 21:11 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1048.eqiad.wmnet with OS bullseye
* 21:06 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152801{{!}}Simple summaries survey for English (T389393)]] (duration: 11m 41s)
* 21:05 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1049.eqiad.wmnet with OS bullseye
* 21:04 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1049.eqiad.wmnet with OS bullseye
* 20:59 cjming@deploy1003: cjming, ksarabia: Continuing with sync
* 20:59 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1048.eqiad.wmnet with OS bullseye
* 20:59 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002&quot;
* 20:56 cjming@deploy1003: cjming, ksarabia: Backport for [[gerrit:1152801{{!}}Simple summaries survey for English (T389393)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 20:55 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1152801{{!}}Simple summaries survey for English (T389393)]]
* 20:51 jsn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152797{{!}}Undeploy first set of Patroller Tools surveys (T389401)]] (duration: 12m 55s)
* 20:45 jsn@deploy1003: jsn: Continuing with sync
* 20:41 jsn@deploy1003: jsn: Backport for [[gerrit:1152797{{!}}Undeploy first set of Patroller Tools surveys (T389401)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:38 jsn@deploy1003: Started scap sync-world: Backport for [[gerrit:1152797{{!}}Undeploy first set of Patroller Tools surveys (T389401)]]
* 20:36 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152165{{!}}Remove wgParserEnableLegacyHeadingDOM option (T371756)]] (duration: 10m 37s)
* 20:29 arlolra@deploy1003: arlolra: Continuing with sync
* 20:27 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1028
* 20:27 arlolra@deploy1003: arlolra: Backport for [[gerrit:1152165{{!}}Remove wgParserEnableLegacyHeadingDOM option (T371756)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:27 jhathaway@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on mx-in2001.wikimedia.org with reason: [[phab:T395240|T395240]]
* 20:26 jhathaway@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on mx-in1001.wikimedia.org with reason: [[phab:T395240|T395240]]
* 20:26 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1028
* 20:25 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1152165{{!}}Remove wgParserEnableLegacyHeadingDOM option (T371756)]]
* 20:23 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152779{{!}}ext.xLab: Send limited copies of stream configs (T391988)]] (duration: 15m 51s)
* 20:22 jhathaway@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on mx-out1001.wikimedia.org with reason: [[phab:T395240|T395240]]
* 20:18 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002&quot;
* 20:16 cjming@deploy1003: cjming, phuedx: Continuing with sync
* 20:16 jhathaway@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on mx-out2001.wikimedia.org with reason: [[phab:T395240|T395240]]
* 20:10 cjming@deploy1003: cjming, phuedx: Backport for [[gerrit:1152779{{!}}ext.xLab: Send limited copies of stream configs (T391988)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:07 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1152779{{!}}ext.xLab: Send limited copies of stream configs (T391988)]]
* 19:40 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 19:40 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 19:36 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1048.eqiad.wmnet with reason: host reimage
* 19:32 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1048.eqiad.wmnet with reason: host reimage
* 19:20 sukhe@cumin1002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restarting P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3008.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:20 sukhe@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3008.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:20 sukhe@cumin1002: START - Cookbook sre.loadbalancer.admin pooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3008.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:20 sukhe@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3008.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:20 sukhe@cumin1002: START - Cookbook sre.loadbalancer.admin depooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3008.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:19 sukhe@cumin1002: START - Cookbook sre.loadbalancer.upgrade restarting P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3008.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:15 sukhe@cumin1002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restarting P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3009.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:15 sukhe@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3009.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:14 sukhe@cumin1002: START - Cookbook sre.loadbalancer.admin pooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3009.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:14 sukhe@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3009.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:14 sukhe@cumin1002: START - Cookbook sre.loadbalancer.admin depooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3009.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:14 sukhe@cumin1002: START - Cookbook sre.loadbalancer.upgrade restarting P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3009.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:08 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1048.eqiad.wmnet with OS bullseye
* 19:06 sukhe@cumin1002: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restarting P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3010.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:06 sukhe@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3010.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:05 sukhe@cumin1002: START - Cookbook sre.loadbalancer.admin pooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3010.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:05 sukhe@cumin1002: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3010.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:05 sukhe@cumin1002: START - Cookbook sre.loadbalancer.admin depooling P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3010.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 19:05 sukhe@cumin1002: START - Cookbook sre.loadbalancer.upgrade restarting P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs3010.esams.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:liberica
* 18:46 jasmine@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wikikube-worker1028.eqiad.wmnet
* 18:46 jasmine@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:44 jasmine@cumin1002: START - Cookbook sre.dns.netbox
* 18:34 jasmine@cumin1002: START - Cookbook sre.hosts.decommission for hosts wikikube-worker1028.eqiad.wmnet
* 18:33 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1104.eqiad.wmnet,service=(cdn{{!}}ats-be)
* 18:32 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76866 and previous config saved to /var/cache/conftool/dbconfig/20250602-183230-fceratto.json
* 18:24 brett: include libvmod-wmfuniq 0.2.0~deb11u1 in bullseye-wikimedia
* 18:23 brett: include libvmod-wmfuniq 0.2.0~deb12u1 in bookworm-wikimedia
* 18:21 ebernhardson@deploy1003: Finished deploy [airflow-dags/search@443d0ab]: bump glent to 0.3.6 (duration: 00m 29s)
* 18:21 ebernhardson@deploy1003: Started deploy [airflow-dags/search@443d0ab]: bump glent to 0.3.6
* 18:17 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P76865 and previous config saved to /var/cache/conftool/dbconfig/20250602-181722-fceratto.json
* 18:10 jasmine@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wikikube-worker[1026-1028].eqiad.wmnet
* 18:10 jasmine@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:10 jasmine@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: wikikube-worker[1026-1028].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jasmine@cumin1002&quot;
* 18:07 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1048.eqiad.wmnet with OS bullseye
* 18:06 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1048.eqiad.wmnet with OS bullseye
* 18:05 jasmine@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: wikikube-worker[1026-1028].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jasmine@cumin1002&quot;
* 18:02 jasmine@cumin1002: START - Cookbook sre.dns.netbox
* 18:02 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P76864 and previous config saved to /var/cache/conftool/dbconfig/20250602-180216-fceratto.json
* 17:50 jasmine@cumin1002: START - Cookbook sre.hosts.decommission for hosts wikikube-worker[1026-1028].eqiad.wmnet
* 17:47 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76863 and previous config saved to /var/cache/conftool/dbconfig/20250602-174708-fceratto.json
* 17:38 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76862 and previous config saved to /var/cache/conftool/dbconfig/20250602-173850-fceratto.json
* 17:38 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 17:33 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 17:33 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76861 and previous config saved to /var/cache/conftool/dbconfig/20250602-173316-fceratto.json
* 17:22 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1048.eqiad.wmnet with OS bullseye
* 17:18 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20250602-171804-fceratto.json
* 17:05 jasmine@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1026-1028].eqiad.wmnet
* 17:04 jasmine@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1026-1028].eqiad.wmnet
* 17:02 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P76860 and previous config saved to /var/cache/conftool/dbconfig/20250602-170256-fceratto.json
* 16:50 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 16:49 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcontrol2010-dev.codfw.wmnet with OS bookworm
* 16:47 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76859 and previous config saved to /var/cache/conftool/dbconfig/20250602-164748-fceratto.json
* 16:44 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcontrol2010-dev.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 16:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host cloudcontrol2010-dev.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 16:40 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76857 and previous config saved to /var/cache/conftool/dbconfig/20250602-164030-fceratto.json
* 16:40 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 16:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcontrol2010-dev.codfw.wmnet with OS bookworm
* 16:40 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76856 and previous config saved to /var/cache/conftool/dbconfig/20250602-164003-fceratto.json
* 16:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 16:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
* 16:30 marostegui@cumin1002: dbctl commit (dc=all): 'db1211 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76855 and previous config saved to /var/cache/conftool/dbconfig/20250602-162957-root.json
* 16:24 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P76854 and previous config saved to /var/cache/conftool/dbconfig/20250602-162455-fceratto.json
* 16:22 sukhe: sudo cumin -b1 -s60 'A:cp and not P&lt;nowiki&gt;{&lt;/nowiki&gt;cp7001*&lt;nowiki&gt;}&lt;/nowiki&gt;' &quot;depool cdn &amp;&amp; sleep 10 &amp;&amp; run-puppet-agent --enable 'merging CR 1091330' &amp;&amp; systemctl restart trafficserver.service &amp;&amp; sleep 10 &amp;&amp; pool cdn&quot;
* 16:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1211 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76853 and previous config saved to /var/cache/conftool/dbconfig/20250602-161452-root.json
* 16:09 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P76852 and previous config saved to /var/cache/conftool/dbconfig/20250602-160948-fceratto.json
* 16:03 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7001.magru.wmnet [reason: [end] testing CR {{Gerrit|1091330}}]
* 15:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1211 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76851 and previous config saved to /var/cache/conftool/dbconfig/20250602-155946-root.json
* 15:55 sukhe: enable puppet and run agent on cp7001
* 15:54 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76850 and previous config saved to /var/cache/conftool/dbconfig/20250602-155441-fceratto.json
* 15:52 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7001.magru.wmnet [reason: testing CR {{Gerrit|1091330}}]
* 15:50 sukhe: disable puppet on A:cp to merge CR: {{Gerrit|1091330}}
* 15:49 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152757{{!}}Enable MetricsPlatform's experimentation feature]] (duration: 14m 23s)
* 15:47 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 15:47 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76849 and previous config saved to /var/cache/conftool/dbconfig/20250602-154734-fceratto.json
* 15:47 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 15:47 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 15:47 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76848 and previous config saved to /var/cache/conftool/dbconfig/20250602-154709-fceratto.json
* 15:46 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-be2009.codfw.wmnet with reason: host reimage
* 15:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1211 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76847 and previous config saved to /var/cache/conftool/dbconfig/20250602-154440-root.json
* 15:42 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be2009.codfw.wmnet with reason: host reimage
* 15:42 phuedx@deploy1003: phuedx: Continuing with sync
* 15:38 phuedx@deploy1003: phuedx: Backport for [[gerrit:1152757{{!}}Enable MetricsPlatform's experimentation feature]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:35 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1152757{{!}}Enable MetricsPlatform's experimentation feature]]
* 15:32 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P76846 and previous config saved to /var/cache/conftool/dbconfig/20250602-153201-fceratto.json
* 15:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1211 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76845 and previous config saved to /var/cache/conftool/dbconfig/20250602-152935-root.json
* 15:27 joal@deploy1003: Finished deploy [airflow-dags/analytics@03db055]: Regular analytics weekly train (with pull...) [airflow-dags/analytics_test@03db0552] (duration: 00m 42s)
* 15:26 joal@deploy1003: Started deploy [airflow-dags/analytics@03db055]: Regular analytics weekly train (with pull...) [airflow-dags/analytics_test@03db0552]
* 15:21 thcipriani: jouncebot nowandnext
* 15:19 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcontrol2010-dev.codfw.wmnet with OS bookworm
* 15:16 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P76844 and previous config saved to /var/cache/conftool/dbconfig/20250602-151654-fceratto.json
* 15:15 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host thanos-be2009.codfw.wmnet with OS bullseye
* 15:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1211 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76843 and previous config saved to /var/cache/conftool/dbconfig/20250602-151429-root.json
* 15:03 joal@deploy1003: Finished deploy [airflow-dags/analytics_test@4ebb376]: Regular analytics weekly train [airflow-dags/analytics_test@4ebb376f] (duration: 00m 05s)
* 15:03 joal@deploy1003: Started deploy [airflow-dags/analytics_test@4ebb376]: Regular analytics weekly train [airflow-dags/analytics_test@4ebb376f]
* 15:03 joal@deploy1003: Finished deploy [airflow-dags/analytics@afad011]: Regular analytics weekly train [airflow-dags/main@afad011c] (duration: 00m 07s)
* 15:03 joal@deploy1003: Started deploy [airflow-dags/analytics@afad011]: Regular analytics weekly train [airflow-dags/main@afad011c]
* 15:01 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76842 and previous config saved to /var/cache/conftool/dbconfig/20250602-150146-fceratto.json
* 15:00 phuedx: Disabled the SDS 2.4.11 Synthetic A/A Test in xLab
* 14:54 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76841 and previous config saved to /var/cache/conftool/dbconfig/20250602-145443-fceratto.json
* 14:54 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 14:54 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76840 and previous config saved to /var/cache/conftool/dbconfig/20250602-145418-fceratto.json
* 14:54 joal@deploy1003: Finished deploy [analytics/refinery@b1aa837] (hadoop-test): Regular analytics weekly train test [analytics/refinery@b1aa837f] (duration: 09m 27s)
* 14:44 joal@deploy1003: Started deploy [analytics/refinery@b1aa837] (hadoop-test): Regular analytics weekly train test [analytics/refinery@b1aa837f]
* 14:44 joal@deploy1003: Finished deploy [analytics/refinery@b1aa837] (thin): Regular analytics weekly train THIN [analytics/refinery@b1aa837f] (duration: 01m 06s)
* 14:43 joal@deploy1003: Started deploy [analytics/refinery@b1aa837] (thin): Regular analytics weekly train THIN [analytics/refinery@b1aa837f]
* 14:42 joal@deploy1003: Finished deploy [analytics/refinery@b1aa837]: Regular analytics weekly train [analytics/refinery@b1aa837f] (duration: 03m 08s)
* 14:39 joal@deploy1003: Started deploy [analytics/refinery@b1aa837]: Regular analytics weekly train [analytics/refinery@b1aa837f]
* 14:39 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P76835 and previous config saved to /var/cache/conftool/dbconfig/20250602-143910-fceratto.json
* 14:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[1154,1211].eqiad.wmnet with reason: Maintenance
* 14:35 bking@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts elastic1067.eqiad.wmnet
* 14:35 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:35 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: elastic1067.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2002&quot;
* 14:35 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: elastic1067.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2002&quot;
* 14:24 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P76833 and previous config saved to /var/cache/conftool/dbconfig/20250602-142403-fceratto.json
* 14:08 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76832 and previous config saved to /var/cache/conftool/dbconfig/20250602-140854-fceratto.json
* 14:04 phuedx: Enabling the SDS 2.4.11 Synthetic A/A Test in xLab
* 13:59 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76831 and previous config saved to /var/cache/conftool/dbconfig/20250602-135945-fceratto.json
* 13:59 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 13:59 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76830 and previous config saved to /var/cache/conftool/dbconfig/20250602-135920-fceratto.json
* 13:52 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:49 jmm@cumin1003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host prometheus7002.magru.wmnet
* 13:49 jmm@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host prometheus7002.magru.wmnet with OS bookworm
* 13:44 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P76829 and previous config saved to /var/cache/conftool/dbconfig/20250602-134413-fceratto.json
* 13:43 bking@cumin2002: START - Cookbook sre.dns.netbox
* 13:41 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1048.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:41 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:41 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:41 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1049.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:38 bking@cumin2002: START - Cookbook sre.hosts.decommission for hosts elastic1067.eqiad.wmnet
* 13:37 bking@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cirrussearch[1064-1066].eqiad.wmnet
* 13:37 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:37 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cirrussearch[1064-1066].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2002&quot;
* 13:37 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cirrussearch[1064-1066].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2002&quot;
* 13:34 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host thanos-be2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:29 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P76828 and previous config saved to /var/cache/conftool/dbconfig/20250602-132906-fceratto.json
* 13:24 Lucas_WMDE: UTC afternoon backport+config window done
* 13:22 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance
* 13:22 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152191{{!}}core-Namespaces: Add Page, Author to default search ns in ruwikisource (T395632)]] (duration: 12m 00s)
* 13:21 bking@cumin2002: START - Cookbook sre.dns.netbox
* 13:20 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1048.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudcephosd1049.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 13:19 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
* 13:17 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: added mgmt for cloudcephosd1048-51  - jclark@cumin1002&quot;
* 13:16 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: added mgmt for cloudcephosd1048-51  - jclark@cumin1002&quot;
* 13:15 lucaswerkmeister-wmde@deploy1003: bunnypranav, lucaswerkmeister-wmde: Continuing with sync
* 13:14 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76826 and previous config saved to /var/cache/conftool/dbconfig/20250602-131359-fceratto.json
* 13:13 lucaswerkmeister-wmde@deploy1003: bunnypranav, lucaswerkmeister-wmde: Backport for [[gerrit:1152191{{!}}core-Namespaces: Add Page, Author to default search ns in ruwikisource (T395632)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 jclark@cumin1002: START - Cookbook sre.dns.netbox
* 13:10 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1152191{{!}}core-Namespaces: Add Page, Author to default search ns in ruwikisource (T395632)]]
* 13:09 bking@cumin2002: START - Cookbook sre.hosts.decommission for hosts cirrussearch[1064-1066].eqiad.wmnet
* 13:05 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76825 and previous config saved to /var/cache/conftool/dbconfig/20250602-130548-fceratto.json
* 13:05 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 13:05 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76824 and previous config saved to /var/cache/conftool/dbconfig/20250602-130523-fceratto.json
* 12:50 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P76823 and previous config saved to /var/cache/conftool/dbconfig/20250602-125016-fceratto.json
* 12:37 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of es2036.codfw.wmnet onto es2047.codfw.wmnet
* 12:37 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) es2047 gradually with 4 steps - Pool es2047.codfw.wmnet in after cloning
* 12:37 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org
* 12:35 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P76821 and previous config saved to /var/cache/conftool/dbconfig/20250602-123508-fceratto.json
* 12:33 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org
* 12:26 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host prometheus7002.magru.wmnet with OS bookworm
* 12:25 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM prometheus7002.magru.wmnet - jmm@cumin1003&quot;
* 12:25 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM prometheus7002.magru.wmnet - jmm@cumin1003&quot;
* 12:24 jmm@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) prometheus7002.magru.wmnet on all recursors
* 12:24 jmm@cumin1003: START - Cookbook sre.dns.wipe-cache prometheus7002.magru.wmnet on all recursors
* 12:24 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:24 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM prometheus7002.magru.wmnet - jmm@cumin1003&quot;
* 12:22 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM prometheus7002.magru.wmnet - jmm@cumin1003&quot;
* 12:20 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76819 and previous config saved to /var/cache/conftool/dbconfig/20250602-122001-fceratto.json
* 12:17 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 12:17 jmm@cumin1003: START - Cookbook sre.ganeti.makevm for new host prometheus7002.magru.wmnet
* 12:12 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh7003.wikimedia.org
* 12:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host doh7003.wikimedia.org with OS bookworm
* 12:11 claime: cgoubert@mwmaint1002:~$ sudo systemctl restart mediawiki_job_generatecaptcha.service  - [[phab:T388531|T388531]]
* 12:11 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet
* 12:10 mvernon@cumin1002: END (PASS) - Cookbook sre.swift.check-dbs (exit_code=0) Checking container DBs of global-data-captcha-render
* 12:10 mvernon@cumin1002: START - Cookbook sre.swift.check-dbs Checking container DBs of global-data-captcha-render
* 12:10 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1182 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76818 and previous config saved to /var/cache/conftool/dbconfig/20250602-121041-fceratto.json
* 12:10 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance
* 12:10 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76817 and previous config saved to /var/cache/conftool/dbconfig/20250602-121016-fceratto.json
* 12:07 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet
* 12:00 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2002.codfw.wmnet
* 11:59 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
* 11:57 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'readability' for release 'main' .
* 11:55 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' .
* 11:55 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P76814 and previous config saved to /var/cache/conftool/dbconfig/20250602-115509-fceratto.json
* 11:54 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host build2002.codfw.wmnet
* 11:51 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on doh7003.wikimedia.org with reason: host reimage
* 11:49 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on doh7003.wikimedia.org with reason: host reimage
* 11:48 marostegui@cumin1002: START - Cookbook sre.mysql.pool es2047 gradually with 4 steps - Pool es2047.codfw.wmnet in after cloning
* 11:47 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
* 11:46 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw1001.wikimedia.org
* 11:42 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ldap-rw1001.wikimedia.org
* 11:40 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P76812 and previous config saved to /var/cache/conftool/dbconfig/20250602-114001-fceratto.json
* 11:39 claime: cgoubert@mwmaint1002:~$ sudo systemctl restart mediawiki_job_generatecaptcha.service  - [[phab:T388531|T388531]]
* 11:37 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw2001.wikimedia.org
* 11:33 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ldap-rw2001.wikimedia.org
* 11:32 claime: Manual run of cronjobs/generatecaptcha on k8s - [[phab:T388531|T388531]]
* 11:31 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
* 11:30 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
* 11:24 fceratto@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76811 and previous config saved to /var/cache/conftool/dbconfig/20250602-112453-fceratto.json
* 11:23 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint1001.eqiad.wmnet
* 11:19 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ldap-maint1001.eqiad.wmnet
* 11:18 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host doh7003.wikimedia.org with OS bookworm
* 11:17 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM doh7003.wikimedia.org - jmm@cumin1003&quot;
* 11:17 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM doh7003.wikimedia.org - jmm@cumin1003&quot;
* 11:17 jmm@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) doh7003.wikimedia.org on all recursors
* 11:17 jmm@cumin1003: START - Cookbook sre.dns.wipe-cache doh7003.wikimedia.org on all recursors
* 11:17 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:17 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM doh7003.wikimedia.org - jmm@cumin1003&quot;
* 11:16 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM doh7003.wikimedia.org - jmm@cumin1003&quot;
* 11:15 fceratto@cumin1002: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T395241|T395241]])', diff saved to https://phabricator.wikimedia.org/P76810 and previous config saved to /var/cache/conftool/dbconfig/20250602-111519-fceratto.json
* 11:15 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 11:14 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 11:11 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 11:11 jmm@cumin1003: START - Cookbook sre.ganeti.makevm for new host doh7003.wikimedia.org
* 11:10 marostegui@cumin1002: dbctl commit (dc=all): 'es2039 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76809 and previous config saved to /var/cache/conftool/dbconfig/20250602-111044-root.json
* 10:58 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum7003.magru.wmnet
* 10:58 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host durum7003.magru.wmnet with OS bookworm
* 10:58 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint2001.codfw.wmnet
* 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'es2039 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76808 and previous config saved to /var/cache/conftool/dbconfig/20250602-105539-root.json
* 10:54 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ldap-maint2001.codfw.wmnet
* 10:48 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) es2036 gradually with 4 steps - Pool es2036.codfw.wmnet in after cloning
* 10:41 kamila@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: apply
* 10:40 kamila@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: apply
* 10:40 marostegui@cumin1002: dbctl commit (dc=all): 'es2039 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76806 and previous config saved to /var/cache/conftool/dbconfig/20250602-104032-root.json
* 10:40 kamila@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: apply
* 10:40 kamila@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: apply
* 10:37 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on durum7003.magru.wmnet with reason: host reimage
* 10:34 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on durum7003.magru.wmnet with reason: host reimage
* 10:31 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet
* 10:27 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet
* 10:25 marostegui@cumin1002: dbctl commit (dc=all): 'es2039 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76804 and previous config saved to /var/cache/conftool/dbconfig/20250602-102526-root.json
* 10:22 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor2003.codfw.wmnet
* 10:18 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet
* 10:12 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet
* 10:10 marostegui@cumin1002: dbctl commit (dc=all): 'es2039 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76802 and previous config saved to /var/cache/conftool/dbconfig/20250602-101020-root.json
* 10:08 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet
* 10:07 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host durum7003.magru.wmnet with OS bookworm
* 10:06 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM durum7003.magru.wmnet - jmm@cumin1003&quot;
* 10:06 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM durum7003.magru.wmnet - jmm@cumin1003&quot;
* 10:06 jmm@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) durum7003.magru.wmnet on all recursors
* 10:06 jmm@cumin1003: START - Cookbook sre.dns.wipe-cache durum7003.magru.wmnet on all recursors
* 10:06 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:06 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM durum7003.magru.wmnet - jmm@cumin1003&quot;
* 10:06 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM durum7003.magru.wmnet - jmm@cumin1003&quot;
* 10:02 marostegui@cumin1002: START - Cookbook sre.mysql.pool es2036 gradually with 4 steps - Pool es2036.codfw.wmnet in after cloning
* 10:02 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 10:02 jmm@cumin1003: START - Cookbook sre.ganeti.makevm for new host durum7003.magru.wmnet
* 09:59 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet
* 09:59 jmm@cumin1003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ncredir7003.magru.wmnet
* 09:59 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7003.magru.wmnet with OS bookworm
* 09:55 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet
* 09:55 marostegui@cumin1002: dbctl commit (dc=all): 'es2039 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76800 and previous config saved to /var/cache/conftool/dbconfig/20250602-095514-root.json
* 09:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2039.codfw.wmnet with reason: Maintenance
* 09:44 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2039 [[phab:T395647|T395647]]', diff saved to https://phabricator.wikimedia.org/P76798 and previous config saved to /var/cache/conftool/dbconfig/20250602-094402-marostegui.json
* 09:40 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 09:33 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 09:28 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7003.magru.wmnet with reason: host reimage
* 09:27 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 09:24 jmm@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7003.magru.wmnet with reason: host reimage
* 09:22 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 09:13 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard1003.eqiad.wmnet
* 09:10 jelto: update gitlab-settings artifact retention to 6 month - [[phab:T395014|T395014]]
* 09:09 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetboard1003.eqiad.wmnet
* 09:02 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard2003.codfw.wmnet
* 08:58 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ncredir7003.magru.wmnet with OS bookworm
* 08:58 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host puppetboard2003.codfw.wmnet
* 08:51 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM ncredir7003.magru.wmnet - jmm@cumin1003&quot;
* 08:51 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.ganeti.makevm: created new VM ncredir7003.magru.wmnet - jmm@cumin1003&quot;
* 08:51 jmm@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ncredir7003.magru.wmnet on all recursors
* 08:51 jmm@cumin1003: START - Cookbook sre.dns.wipe-cache ncredir7003.magru.wmnet on all recursors
* 08:51 jmm@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:51 jmm@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM ncredir7003.magru.wmnet - jmm@cumin1003&quot;
* 08:51 jmm@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Add records for VM ncredir7003.magru.wmnet - jmm@cumin1003&quot;
* 08:51 marostegui@cumin1002: dbctl commit (dc=all): 'es1040 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76796 and previous config saved to /var/cache/conftool/dbconfig/20250602-085105-root.json
* 08:47 jmm@cumin1003: START - Cookbook sre.dns.netbox
* 08:47 jmm@cumin1003: START - Cookbook sre.ganeti.makevm for new host ncredir7003.magru.wmnet
* 08:45 jmm@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ncredir7003.magru.wmnet with OS bookworm
* 08:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ncredir7003.magru.wmnet
* 08:42 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:42 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ncredir7003.magru.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 08:41 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: ncredir7003.magru.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002&quot;
* 08:37 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 08:36 marostegui@cumin1002: dbctl commit (dc=all): 'es1040 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76795 and previous config saved to /var/cache/conftool/dbconfig/20250602-083559-root.json
* 08:33 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ncredir7003.magru.wmnet
* 08:20 marostegui@cumin1002: dbctl commit (dc=all): 'es1040 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76794 and previous config saved to /var/cache/conftool/dbconfig/20250602-082053-root.json
* 08:11 jmm@cumin1003: START - Cookbook sre.hosts.reimage for host ncredir7003.magru.wmnet with OS bookworm
* 08:05 marostegui@cumin1002: dbctl commit (dc=all): 'es1040 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76793 and previous config saved to /var/cache/conftool/dbconfig/20250602-080547-root.json
* 07:58 phuedx@deploy1003: Finished scap sync-world: Backport for [[gerrit:1152253{{!}}Beta Cluster: Support A/B experiments (T393918)]] (duration: 35m 59s)
* 07:50 marostegui@cumin1002: dbctl commit (dc=all): 'es1040 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76792 and previous config saved to /var/cache/conftool/dbconfig/20250602-075041-root.json
* 07:49 phuedx@deploy1003: phuedx, dr0ptp4kt: Continuing with sync
* 07:38 phuedx@deploy1003: phuedx, dr0ptp4kt: Backport for [[gerrit:1152253{{!}}Beta Cluster: Support A/B experiments (T393918)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:36 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7002.magru.wmnet
* 07:35 marostegui@cumin1002: dbctl commit (dc=all): 'es1040 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76791 and previous config saved to /var/cache/conftool/dbconfig/20250602-073535-root.json
* 07:27 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host ganeti7002.magru.wmnet
* 07:22 phuedx@deploy1003: Started scap sync-world: Backport for [[gerrit:1152253{{!}}Beta Cluster: Support A/B experiments (T393918)]]
* 07:20 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast5004.wikimedia.org
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 6 hosts with reason: Maintenance
* 07:13 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1040.eqiad.wmnet with reason: Maintenance
* 07:08 marostegui@cumin1002: dbctl commit (dc=all): 'db1169 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76790 and previous config saved to /var/cache/conftool/dbconfig/20250602-070837-root.json
* 07:08 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast4005.wikimedia.org
* 07:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1040 [[phab:T395647|T395647]]', diff saved to https://phabricator.wikimedia.org/P76789 and previous config saved to /var/cache/conftool/dbconfig/20250602-070602-marostegui.json
* 07:02 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host bast4005.wikimedia.org
* 06:59 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 06:53 marostegui@cumin1002: dbctl commit (dc=all): 'db1169 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76787 and previous config saved to /var/cache/conftool/dbconfig/20250602-065331-root.json
* 06:53 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 06:52 jmm@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 06:48 jmm@cumin1003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 06:38 marostegui@cumin1002: dbctl commit (dc=all): 'db1169 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76786 and previous config saved to /var/cache/conftool/dbconfig/20250602-063826-root.json
* 06:23 marostegui@cumin1002: dbctl commit (dc=all): 'db1169 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76785 and previous config saved to /var/cache/conftool/dbconfig/20250602-062320-root.json
* 06:08 marostegui@cumin1002: dbctl commit (dc=all): 'db1169 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76783 and previous config saved to /var/cache/conftool/dbconfig/20250602-060815-root.json
* 05:53 marostegui@cumin1002: dbctl commit (dc=all): 'db1169 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76782 and previous config saved to /var/cache/conftool/dbconfig/20250602-055309-root.json
* 05:39 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1169 [[phab:T395663|T395663]]', diff saved to https://phabricator.wikimedia.org/P76781 and previous config saved to /var/cache/conftool/dbconfig/20250602-053905-marostegui.json
* 05:38 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1169.eqiad.wmnet with reason: Maintenance
* 05:20 marostegui@cumin1002: dbctl commit (dc=all): 'Add es2047 to dbctl depooled [[phab:T395771|T395771]]', diff saved to https://phabricator.wikimedia.org/P76780 and previous config saved to /var/cache/conftool/dbconfig/20250602-051957-marostegui.json
* 05:15 marostegui@cumin1002: START - Cookbook sre.mysql.clone of es2036.codfw.wmnet onto es2047.codfw.wmnet
* 05:02 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 05:01 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2036 [[phab:T395771|T395771]]', diff saved to https://phabricator.wikimedia.org/P76779 and previous config saved to /var/cache/conftool/dbconfig/20250602-050150-marostegui.json

==Archives ==
See [[Server Admin Log/Archives]].
&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>hr8tush4ari9615doj18zd6yua28rfs</sha1>
    </revision>
  </page>
  <page>
    <title>User talk:Inductiveload</title>
    <ns>3</ns>
    <id>9814</id>
    <revision>
      <id>2309612</id>
      <parentid>2066157</parentid>
      <timestamp>2025-06-07T18:22:30Z</timestamp>
      <contributor>
        <username>Alien333</username>
        <id>39666</id>
      </contributor>
      <comment>/* Intend to adopt pagelister */ new section</comment>
      <origin>2309612</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1895" sha1="scbm2u7i149hd08h66qdyoc4w19khhx" xml:space="preserve">== Your shell access was granted ==

Hello, welcome to Wikimedia Labs! I would like to inform you that your shell request was processed and you should be able to login to Bastion host now at bastion.wmflabs.org. In case you get into troubles, please read [[Help:Access|this page]] which contains useful information about accessing instances. You can also ask in our irc channel at {{irc|wikimedia-labs}} or send an e-mail to our mailing list labs-l@lists.wikimedia.org - thank you, and have fun using labs! [[User:Tim Landscheidt|Tim Landscheidt]] ([[User talk:Tim Landscheidt|talk]]) 02:56, 5 December 2013 (UTC)
== Welcome to Tool Labs! ==
Hello Inductiveload, welcome to the Tool Labs project!  Your request for access was processed and you should be able to log in now with &lt;code&gt;ssh&lt;/code&gt; at &lt;code&gt;login.tools.wmflabs.org&lt;/code&gt;.  To get started, check the [[Help:Tool Labs|help page]].  You can also ask in our IRC channel at {{irc|wikimedia-cloud}} or send an e-mail to our mailing list &lt;code&gt;labs-l@lists.wikimedia.org&lt;/code&gt; – thank you, and have fun using Tools! --[[User:StrikerBot|StrikerBot]] ([[User talk:StrikerBot|talk]]) 19:51, 24 July 2017 (UTC)

== il-general review ==

Hi, we're doing a [[Portal:Toolforge/Tool_sweep | review]] to make sure as many tools meet a minimum standard. I noticed [https://toolsadmin.wikimedia.org/tools/id/Inductiveload il-general] doesn't have a published repo. Could you add this information to this tool? Thanks! [[User:Komla Sapaty|Komla Sapaty]] ([[User talk:Komla Sapaty|talk]]) 21:48, 3 April 2023 (UTC)

== Intend to adopt pagelister ==

Hello. Given you're not there anymore (and good luck with whatever you're doing these days), and pagelister is broken and needs a line changed (IIIF url changed), I intend to request adoption of that tool in 14 days. [[User:Alien333|Alien333]] ([[User talk:Alien333|talk]]) 18:22, 7 June 2025 (UTC)</text>
      <sha1>scbm2u7i149hd08h66qdyoc4w19khhx</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Admin/SAL</title>
    <ns>498</ns>
    <id>30942</id>
    <revision>
      <id>2309614</id>
      <parentid>2309588</parentid>
      <timestamp>2025-06-07T18:44:27Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova</comment>
      <origin>2309614</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1268360" sha1="m9llsp68kwg9f0nhxh6dhyd01jk5bu9" xml:space="preserve">=== 2025-06-07 ===
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova

=== 2025-06-06 ===
* 20:59 andrewbogott: restarting all designate services on all cloudcontrols in eqiad1

=== 2025-06-05 ===
* 20:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,octavia
* 15:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:52 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2025-06-03 ===
* 22:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 20:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:44 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:43 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 17:50 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:36 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-06-02 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 21:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237
* 21:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242
* 17:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 15:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 00:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T394333|T394333]])

=== 2025-06-01 ===
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T394333|T394333]])

=== 2025-05-31 ===
* 23:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services ([[phab:T395742|T395742]])
* 23:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services ([[phab:T395742|T395742]])
* 23:38 andrewbogott: failing over from cloudnet1005 to 1006 in hopes of unsticking [[phab:T395742|T395742]]
* 23:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,neutron ([[phab:T395742|T395742]])
* 23:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,neutron ([[phab:T395742|T395742]])

=== 2025-05-30 ===
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-28 ===
* 22:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1076.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1076.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1075.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1075.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1074.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1074.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1073.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1073.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1072.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1072.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1071.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1071.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1070.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1070.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1069.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1069.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1068.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1068.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudrabbot1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbot1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,designate
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate

=== 2025-05-26 ===
* 13:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet2006-dev.codfw.wmnet
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet2006-dev.codfw.wmnet
* 13:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet2005-dev.codfw.wmnet
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet2005-dev.codfw.wmnet
* 12:54 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=99)
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.roll_reboot_cloudnets
* 12:49 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=99)
* 12:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.roll_reboot_cloudnets
* 12:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::control&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::control&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 07:52 taavi: add new /dev/sdb back to software raid after it was replaced in cloudcephmon1004

=== 2025-05-23 ===
* 09:00 dhinus: failover dumps_dist_active_vps to clouddumps1001 ([[phab:T383723|T383723]])

=== 2025-05-20 ===
* 19:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 15:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 14:57 andrewbogott: resetting eqiad1 rabbitmq in an attempt to resolve [[phab:T394790|T394790]]
* 03:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 03:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 02:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 00:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 00:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])

=== 2025-05-19 ===
* 22:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 22:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 22:43 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for service: project,nova
* 22:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova
* 22:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:26 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for service: project,nova
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 18:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T394727|T394727]])
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1075.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1075.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1072.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1072.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1076.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1076.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1073.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1073.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-15 ===
* 15:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 14:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 13:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-14 ===
* 17:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 17:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/231
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/231
* 12:59 taavi: powercycle unresponsive cloudnet2006-dev

=== 2025-05-12 ===
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 08:17 taavi: powercycle clouservices2005-dev.codfw.wmnet
* 03:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 03:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 02:36 andrewbogott: rebooting cloudnet2005-dev from mgmt  -- ssh is failing and the console shows a user prompt but not a password prompt.

=== 2025-05-11 ===
* 13:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova

=== 2025-05-07 ===
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 19:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:19 andrewbogott: upgrading codfw1dev to version 'epoxy' [[phab:T390914|T390914]]
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T390914|T390914]])
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T390914|T390914]])
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum,heat
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum,heat
* 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum,heat
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum,heat
* 16:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum
* 16:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,nova
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 13:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,neutron
* 13:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,neutron
* 13:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,neutron
* 13:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,neutron
* 12:48 taavi: updating all security group rules referencing old 172.16.0.0/21 subnet to reference new ip space instead ([[phab:T379175|T379175]])
* 10:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-05-06 ===
* 03:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)

=== 2025-05-05 ===
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 03:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 00:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 00:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 00:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-05-04 ===
* 23:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:53 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:41 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2025-05-03 ===
* 02:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])

=== 2025-05-02 ===
* 23:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 21:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 16:20 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=97) ([[phab:T393196|T393196]])
* 16:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 16:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 16:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrewbogott: sudo cumin --force O&lt;nowiki&gt;{&lt;/nowiki&gt;*&lt;nowiki&gt;}&lt;/nowiki&gt; &quot;dpkg --list {{!}} grep puppetserver &amp;&amp; systemctl restart puppetserver.service&quot; # work around a package update that is causing some puppetservers to error out

=== 2025-04-29 ===
* 13:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225
* 12:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225
* 11:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:55 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223
* 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223
* 07:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 07:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221

=== 2025-04-28 ===
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:21 taavi: migrating opentofu managed default security group rules [[phab:T392799|T392799]]
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-26 ===
* 07:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0) ([[phab:T390134|T390134]])
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance ([[phab:T390134|T390134]])
* 07:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T390134|T390134]])
* 07:47 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T390134|T390134]])

=== 2025-04-25 ===
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218
* 12:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 04:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T390134|T390134]])

=== 2025-04-24 ===
* 17:25 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T390134|T390134]])
* 13:29 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for service: project,designate
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate

=== 2025-04-23 ===
* 21:45 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 21:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:45 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:21 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 19:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:32 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:29 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:29 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:17 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:27 arturo: enabling IPv6 dualstack on neutron virtual router ([[phab:T380174|T380174]])
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:14 arturo: enable IPv6 on cloudgw ([[phab:T380174|T380174]]) -- includes server reboot

=== 2025-04-22 ===
* 04:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)

=== 2025-04-21 ===
* 23:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 23:34 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 23:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 23:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 22:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:09 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 22:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 16:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 01:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 01:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-16 ===
* 16:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 arturo: merging network change in neutron https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/198
* 10:10 dcaro: upgrade spicerack on cloudcumin2001 to 10.1.0
* 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 10:07 dcaro: upgrade spicerack on cloudcumin1001 to 10.1.0

=== 2025-04-15 ===
* 15:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:38 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:31 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:30 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-14 ===
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 17:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:50 andrewbogott: granting 'tofuadmin' user inherited 'member' role in all projects, this should fix some policy mishaps
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-13 ===
* 22:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 22:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-11 ===
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-10 ===
* 12:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:05 arturo: [codfw1dev] root@cloudcontrol2004-dev:~# wmcs-makedomain --project testlabs --domain testlabs.codfw1dev.wmcloud.org --orig-project cloudinfra-codfw1dev ([[phab:T391325|T391325]])

=== 2025-04-09 ===
* 23:23 bd808: Rebooting tools-sgebastion-10 (login-buster.toolforge.org) for high load/unresponsive NFS mounts ([[phab:T391538|T391538]])
* 10:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 08:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 04:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 03:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:30 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 03:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services

=== 2025-04-08 ===
* 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet'
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet'
* 22:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1011.eqiad.wmnet'
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet'
* 11:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2025-04-07 ===
* 23:43 bd808: `sudo service maintain-dbusers restart` on cloudcontrol1007 after reports of missing replica.my.cnf and finding the journal for the service empty.
* 15:30 arturo: [codfw1dev] testlabs create a bunch of VMs by hand, like `networktests-vlan-legacy-floating` [[phab:T380728|T380728]]
* 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:23 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-06 ===
* 02:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 02:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-04 ===
* 15:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 05:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 05:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:10 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for all services
* 05:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:04 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for all services
* 05:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-03 ===
* 22:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 20:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 20:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 20:11 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:29 taavi: run wmcs-wikireplica-dns to create transitional x3 CNAMEs [[phab:T390954|T390954]]

=== 2025-04-02 ===
* 20:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup2004.codfw.wmnet' ([[phab:T381499|T381499]])
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.unset_maintenance (exit_code=0) ({{Gerrit|1133432}})
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.unset_maintenance ({{Gerrit|1133432}})
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:22 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:26 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T381499|T381499]])
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T381499|T381499]])
* 09:53 dhinus: systemctl restart maintain-dbusers.service (attempting to fix a weird issue)

=== 2025-04-01 ===
* 21:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate
* 21:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 21:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 21:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 21:00 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-31 ===
* 15:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 15:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 15:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 15:53 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2025-03-27 ===
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 14:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 12:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T390134|T390134]])

=== 2025-03-26 ===
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-25 ===
* 14:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-24 ===
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-22 ===
* 03:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,designate
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate

=== 2025-03-19 ===
* 14:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,designate
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 12:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,task_id,no_dologmsg,cluster_name,all_services,nova,glance,keystone,cinder,neutron,trove,magnum,heat,swift,designate,filter_nodes
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,task_id,no_dologmsg,cluster_name,all_services,nova,glance,keystone,cinder,neutron,trove,magnum,heat,swift,designate,filter_nodes
* 06:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 06:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 06:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 06:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet'
* 06:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 06:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet'
* 06:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet'
* 06:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 06:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet'
* 06:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 06:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 05:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet'
* 05:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 05:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 05:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 05:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 04:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 04:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 04:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet'
* 04:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 04:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 04:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet'
* 04:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet'
* 04:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet'
* 04:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet'
* 04:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet'
* 04:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 03:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T381499|T381499]])
* 02:12 andrewbogott: upgrading codfw1dev to openstack 'dalmation'  [[phab:T381499|T381499]]
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T381499|T381499]])

=== 2025-03-14 ===
* 13:36 volans: installed cumin v5.1.1 on cloudcumin* hosts

=== 2025-03-05 ===
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 00:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 00:43 andrewbogott: restarting all openstack services in hopes of gettting dns unstuck
* 00:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services

=== 2025-03-04 ===
* 08:52 arturo: [[phab:T387828|T387828]] depooled galera on cloudcontrol1005

=== 2025-03-01 ===
* 19:35 andrewbogott: installed new bookworm base images in eqiad1
* 19:35 andrewbogott: installed new bookworm and bullseye base images in codfw1dev

=== 2025-02-28 ===
* 15:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-27 ===
* 15:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-24 ===
* 00:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2025-02-23 ===
* 21:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-21 ===
* 12:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-20 ===
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T386083|T386083]])
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T386083|T386083]])

=== 2025-02-19 ===
* 13:26 arturo: manual failover of cloudgw1004 to cloudgw1003 [[phab:T382356|T382356]]
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-18 ===
* 12:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-02-13 ===
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-11 ===
* 11:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T386083|T386083]])
* 11:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T386083|T386083]])
* 11:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047' ([[phab:T386083|T386083]])
* 11:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047' ([[phab:T386083|T386083]])

=== 2025-02-07 ===
* 13:52 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet1005.eqiad.wmnet ([[phab:T384946|T384946]])
* 13:48 root@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet1005.eqiad.wmnet ([[phab:T384946|T384946]])
* 02:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 01:48 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2025-02-06 ===
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:45 andrewbogott: cold-migrating all remaining VMs in [[phab:T385264|T385264]] except for 'integration' and 'tools' VMs

=== 2025-02-05 ===
* 19:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 19:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2025-02-04 ===
* 12:48 arturo: replacing cloudgw1002 with cloudgw1004 - [[phab:T382356|T382356]]
* 09:22 arturo: fleet-wide restart of puppetservers [[phab:T385553|T385553]]

=== 2025-02-03 ===
* 13:42 andrewbogott: rebooting proxy-04.project-proxy for the ceph OSD mishap

=== 2025-02-02 ===
* 17:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 17:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2025-01-31 ===
* 01:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-01-30 ===
* 21:39 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 21:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:33 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 12:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1031.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1031.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 03:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 02:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 02:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 00:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 00:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)

=== 2025-01-29 ===
* 23:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 23:43 andrewbogott: resetting rabbitmq in eqiad1 in hopes that it will resolve mysterious openstack misbehaviors
* 20:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2009-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2009-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1007.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1007.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1005.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1005.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-28 ===
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:51 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 13:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 13:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 13:02 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T348643|T348643]])
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 00:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 00:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T348643|T348643]])
* 00:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2025-01-27 ===
* 21:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:52 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 14:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-24 ===
* 13:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node

=== 2025-01-23 ===
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 20:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 15:49 dhinus: cumin 'P:base::cloud_production' 'rm /var/lib/prometheus/node.d/kernel-panic.prom' [[phab:T382961|T382961]]
* 15:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-22 ===
* 14:52 dhinus: cloudcumin[12]001 upgrade spicerack from 8.15.2 to 9.1.0

=== 2025-01-21 ===
* 13:42 andrewbogott: migrating/rebooting VMs as per earlier email,  [[phab:T383583|T383583]]

=== 2025-01-20 ===
* 14:23 dhinus: cumin upgraded form 4.2.0 to 5.0.0 on cloudcumin[12]001. patch [[phab:T346453|T346453]] reapplied.

=== 2025-01-14 ===
* 21:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T383583|T383583]])
* 21:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T383583|T383583]])
* 20:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T383583|T383583]])
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T383583|T383583]])
* 20:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T383583|T383583]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T383583|T383583]])
* 18:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.roll_restart_osd_daemons (exit_code=0)

=== 2025-01-13 ===
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.roll_restart_osd_daemons

=== 2025-01-10 ===
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-01-09 ===
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2025-01-08 ===
* 14:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 13:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2025-01-03 ===
* 21:01 bd808: `sudo service maintain-dbusers restart` on cloudcontrol1005. Report of missing replica.my.cnf and journalctl output empty due to log rotation. ([[phab:T382962|T382962]])

=== 2024-12-18 ===
* 15:33 arturo: cloudgw failover for [[phab:T382220|T382220]]

=== 2024-12-04 ===
* 17:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 17:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 15:09 andrewbogott: rebooted cloudinfra-cloudvps-puppetserver-1, it's so busy that it's unresponsive
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])

=== 2024-11-27 ===
* 13:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet'
* 13:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'

=== 2024-11-26 ===
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 15:16 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:15 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:15 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:15 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:14 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/145
* 15:13 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/145
* 15:12 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:12 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:31 dcaro: added cloudcephmon1004 to the ceph mon pool
* 12:41 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 05:40 andrewbogott: rebooting tools-sgebastion-10.tools.eqiad1.wikimedia.cloud to get NFS things remounted
* 04:56 andrewbogott: 'systemctl restart nfs-server' on tools-nfs-2.tools.eqiad1.wikimedia.cloud

=== 2024-11-25 ===
* 14:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:40 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 13:55 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:28 arturo: create IPv6 networks in eqiad1 ([[phab:T380174|T380174]]), then reverted because network outage
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-23 ===
* 14:58 taavi: removed broken records for re-created VMs in .eqiad.wmflabs zone

=== 2024-11-22 ===
* 14:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:35 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:48 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-20 ===
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2024-11-19 ===
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:43 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:43 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:37 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:35 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 11:39 arturo: [codfw1dev] performing rabbit full reset [[phab:T380208|T380208]]
* 10:26 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:24 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 10:24 arturo: [codfw1dev] restart rabbitmq and nova/neutron services for [[phab:T380208|T380208]]

=== 2024-11-16 ===
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 07:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-15 ===
* 19:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:10 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:30 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:29 arturo: [codfw1dev] restart rabbitmq and designate
* 16:29 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-14 ===
* 09:52 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:48 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-13 ===
* 13:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/120
* 13:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/120

=== 2024-11-11 ===
* 14:49 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-11-08 ===
* 16:47 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:45 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 16:42 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:42 arturo: [codfw1dev] restart all nova services and rabbitmq out of despair
* 16:42 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:45 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:44 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:33 aborrero@cloudcumin2001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 13:32 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:28 arturo: [codfw1dev] restart rabbitmq, openstack services logs show connection errors

=== 2024-11-07 ===
* 17:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 17:42 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-06 ===
* 13:09 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 11:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/116
* 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/116
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-05 ===
* 10:24 arturo: [codfw1dev] disable puppet and make changes for testing [[phab:T378192|T378192]]

=== 2024-11-04 ===
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:09 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:08 aborrero@cloudcumin2001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:08 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 14:08 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:08 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:52 arturo: [codfw1dev] live-hack cloudlb2001-dev and cloudcontrol2004-dev for [[phab:T378192|T378192]]
* 13:52 arturo: [codfw1dev] restart rabbitmq
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-29 ===
* 15:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-28 ===
* 17:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:00 arturo: [codfw1dev] restarting rabbitmq, misbehaving
* 17:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 dhinus: apt full-upgrade and reboot for cloudcumin*
* 16:21 dhinus: upgrade spicerack from 8.8.0 to 8.15.1 on cloudcumin*

=== 2024-10-24 ===
* 15:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-22 ===
* 15:02 taavi: recover access to User:Labslogbot [[phab:T376220|T376220]]

=== 2024-10-21 ===
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:06 arturo: [codfw1dev] restart rabbitmq, tofu shows error talking to the designate API

=== 2024-10-16 ===
* 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-15 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:38 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:33 arturo: cloudgw maintenance, firewall change for [[phab:T374714|T374714]]
* 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-11 ===
* 15:31 arturo: cloudgw maintenance firewall change  [[phab:T374716|T374716]]
* 09:51 arturo: cloudgw network maintenance related to [[phab:T376879|T376879]]
* 08:33 arturo: [codfw1dev] reboot cloudgw2002-dev/2003-dev because network connectivity issues

=== 2024-10-10 ===
* 12:04 arturo: manual network failover in cloudgw because maintenance related to [[phab:T376879|T376879]]
* 10:40 dhinus: cumin 'cloudrabbit*' 'systemctl restart rabbitmq-server' [[phab:T376802|T376802]]
* 09:19 arturo: [codfw1dev] enable IPv6 on cloudgw ([[phab:T374716|T374716]])

=== 2024-10-09 ===
* 14:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:20 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:19 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:19 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:16 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:16 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:58 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:58 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:55 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:55 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:54 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:53 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:53 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95
* 10:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2024-10-08 ===
* 11:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-07 ===
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-04 ===
* 15:12 arturo: cloudservice1005/1006: enable puppet and restore /etc/powerdns/recursor.conf to non-debug mode ([[phab:T374830|T374830]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:32 arturo: cloudservice1005/1006: disable puppet and set `quiet=no` in /etc/powerdns/recursor.conf ([[phab:T374830|T374830]])

=== 2024-10-03 ===
* 14:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:54 arturo: [codfw1dev] delete default security group rule list, now tracking them via tofu-infra
* 14:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:51 arturo: delete default security group rule list, now tracking them via tofu-infra
* 14:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-02 ===
* 15:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:21 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:16 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:15 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:05 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 09:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:10 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])

=== 2024-10-01 ===
* 15:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-30 ===
* 20:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T372814|T372814]])
* 16:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 16:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 13:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 13:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T372814|T372814]])
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 13:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 11:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 11:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 11:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 11:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 10:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 10:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=99)
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=99)
* 09:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 09:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-27 ===
* 15:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:56 arturo: [codfw1dev] restart rabbitmq again
* 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:15 arturo: [codfw1dev] restart rabbitmq
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:04 arturo: [codfw1dev] enable IPv6 on the neutron virtual router [[phab:T375847|T375847]]

=== 2024-09-26 ===
* 14:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 14:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T372814|T372814]])
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 10:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T372814|T372814]])
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:47 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:45 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-25 ===
* 14:11 arturo: [codfw1dev] start proxy-02 vm on proxy-codfw1dev project, it was in shutoff mode for unknown reasons
* 10:33 arturo: [codfw1dev] cleanup unused security groups ([[phab:T375604|T375604]])
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:48 arturo: [codfw1dev] restart rabbitmq on all cloudcontrol servers
* 09:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:35 arturo: [codfw1dev] deletre a bunch of tests and seemingly unused projects

=== 2024-09-24 ===
* 19:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 16:00 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 14:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T348643|T348643]])
* 14:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 14:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:56 arturo: [codfw1dev] restart rabbitmq-server on all 3 nodes
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack
* 09:10 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_rack (exit_code=97)
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack

=== 2024-09-23 ===
* 19:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_rack (exit_code=99)
* 15:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:42 arturo: put cloudvirt1048 in the network-ovs aggregate [[phab:T364457|T364457]]
* 14:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:37 arturo: [codfw1dev] restart rabbitmq
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:23 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-09-21 ===
* 10:17 dhinus: nova host-evacuate cloudvirt1063 ([[phab:T375223|T375223]])
* 09:59 dhinus: openstack aggregate remove host ceph cloudvirt1063 ([[phab:T375223|T375223]])
* 09:59 dhinus: openstack aggregate add host maintenance cloudvirt1063 ([[phab:T375223|T375223]])
* 09:42 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1063.eqiad.wmnet'
* 09:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1063.eqiad.wmnet'

=== 2024-09-20 ===
* 11:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 10:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 08:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T373740|T373740]])
* 08:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T373740|T373740]])

=== 2024-09-19 ===
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T373740|T373740]])
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T373740|T373740]])
* 15:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) ([[phab:T373740|T373740]])
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance ([[phab:T373740|T373740]])
* 10:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51
* 10:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51
* 09:56 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T374043|T374043]])
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 09:51 arturo: [codfw1dev] play with neutron default security group rules (delete, create them, etc) [[phab:T375111|T375111]]
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50

=== 2024-09-18 ===
* 15:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:21 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 15:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/47
* 11:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/47
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:40 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:39 dcaro: restarted rabbitmq-server on all cloudrabbits
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-09-17 ===
* 21:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T374043|T374043]])
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 16:11 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T374043|T374043]])
* 16:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 15:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46

=== 2024-09-16 ===
* 14:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 11:28 arturo: [codfw1dev] created VM bastion-codfw1dev-04 to replace current bastion -03 ([[phab:T374828|T374828]])

=== 2024-09-12 ===
* 10:51 arturo: merging change to keystone wmf hooks https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071230 ([[phab:T374020|T374020]])

=== 2024-09-11 ===
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 16:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 15:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 15:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/41
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/41
* 11:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 07:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T374467|T374467]])
* 07:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T374467|T374467]])

=== 2024-09-10 ===
* 12:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 11:37 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:42 dcaro: hard-rebooting cloudvirt2004-dev (codfw1dev) having io/hardware issues
* 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 08:45 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-09 ===
* 21:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T373986|T373986]])
* 16:36 dcaro: cleaned up dns leaks
* 15:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/39
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/39
* 15:05 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 14:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:06 arturo: merged change to cloudgw NAT setting https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071189
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:42 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:38 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-06 ===
* 23:18 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T373986|T373986]])
* 18:17 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 13:46 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 07:27 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-05 ===
* 21:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 17:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T374043|T374043]])
* 14:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:19 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 14:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 12:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/36
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/36
* 10:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/35
* 10:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/35
* 09:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/34
* 09:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/34
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/33
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/33
* 09:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:22 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/31
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/31
* 09:16 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 08:46 arturo: [codfw1dev] restart rabbitmq @ codfw1dev [[phab:T374002|T374002]]

=== 2024-09-04 ===
* 21:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 19:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:35 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 12:18 arturo: [codfw1dev] restart rabbitmq-server.service on all 3 cloudcontrols, all nova-compute agents are down complaining about rabbitmq being unreachable

=== 2024-09-02 ===
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 11:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28

=== 2024-08-31 ===
* 13:55 andrewbogott: moving tools-redis-7 off of cloudvirt1048 just in case  [[phab:T373740|T373740]]
* 13:39 andrewbogott: rebooting cloudvirt1048 from mgmt, it seems to have crashed

=== 2024-08-30 ===
* 12:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-29 ===
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-08-25 ===
* 22:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-23 ===
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T369044|T369044]])
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T369044|T369044]])
* 00:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-22 ===
* 23:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=97) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.unset_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.unset_maintenance ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=97) on host 'cloudvirt1039' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1037' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1035' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035' ([[phab:T369044|T369044]])
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1033' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1052' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052' ([[phab:T369044|T369044]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])

=== 2024-08-20 ===
* 03:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)

=== 2024-08-19 ===
* 23:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 23:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 23:16 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-08-17 ===
* 03:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-16 ===
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:20 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T363344|T363344]])
* 05:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:57 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:50 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 01:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:12 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-15 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:27 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 08:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 04:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 02:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-14 ===
* 23:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:05 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-12 ===
* 15:44 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 08:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=0)
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:37 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97) ([[phab:T371878|T371878]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) ([[phab:T371878|T371878]])
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) ([[phab:T371878|T371878]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:17 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97)
* 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_rack

=== 2024-08-09 ===
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 18:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 13:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 13:38 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 13:35 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 13:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 11:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 05:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 05:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 00:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 00:47 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])

=== 2024-08-08 ===
* 23:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet'
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet'
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 18:39 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet'
* 18:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet'
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet'
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet'
* 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet'
* 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet'
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet'
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 16:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet'
* 16:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet'
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudbackup1002-dev.codfw.wmnet'
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.codfw.wmnet'
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet'
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet'
* 16:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2006-dev.codfw.wmnet'
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2006-dev.codfw.wmnet'
* 16:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet'
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 15:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 15:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.wikimedia.org'
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.wikimedia.org'
* 15:51 andrewbogott: upgrading codfw1dev to openstack version caracal https://phabricator.wikimedia.org/T369044
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 15:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 15:49 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T369044|T369044]])
* 15:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 14:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 14:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.wait_for_rebalance
* 13:00 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 12:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 12:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 12:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 11:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T371878|T371878]])
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T371878|T371878]])
* 09:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 09:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 07:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 07:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 06:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 05:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 05:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)

=== 2024-08-07 ===
* 20:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 20:11 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=97)
* 20:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 20:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 20:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 18:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 17:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:56 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=97)
* 16:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node
* 16:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node
* 16:40 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:24 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T371878|T371878]])
* 08:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T371878|T371878]])
* 06:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 03:08 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])

=== 2024-08-06 ===
* 21:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 19:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 18:36 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet'
* 18:21 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1042.eqiad.wmnet'
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet'
* 18:17 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet'
* 18:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet'
* 17:59 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet'
* 17:58 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet'
* 17:45 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet'
* 17:43 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet'
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet'
* 17:32 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet'
* 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1045.eqiad.wmnet'
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet'
* 17:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet'
* 17:21 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet'
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet'
* 17:14 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet'
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet'
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 17:01 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet'
* 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 17:00 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:48 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:47 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 16:47 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:37 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 16:37 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:25 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance

=== 2024-08-01 ===
* 13:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-07-31 ===
* 23:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 23:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 16:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27

=== 2024-07-30 ===
* 11:29 arturo: installing nova security updates ([[phab:T371240|T371240]])

=== 2024-07-29 ===
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:28 arturo: [codfw1dev] restarting rabbitmq-server on all cloudcontrols, nova-compute cannot contact it
* 11:00 arturo: [codfw1dev] installing nova security updates ([[phab:T371240|T371240]])
* 10:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27

=== 2024-07-25 ===
* 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:46 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:45 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:09 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:55 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 11:40 arturo: manually restart maintain-dbusers in cloudcontrol1005 to see if that makes any difference
* 11:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 07:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 07:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23

=== 2024-07-24 ===
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 16:01 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:46 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:43 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:37 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 11:59 arturo: restarted maintain-dbusers.service in cloudcontrol1005, it was stuck doing nothing
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:43 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:42 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24

=== 2024-07-23 ===
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 10:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 09:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 05:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 05:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 05:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 05:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 03:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 03:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 02:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 02:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)

=== 2024-07-22 ===
* 23:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 23:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 21:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 20:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 18:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 15:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:28 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 13:28 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:10 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy

=== 2024-07-21 ===
* 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 09:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 06:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 06:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 03:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 03:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 00:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-20 ===
* 21:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 18:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 18:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 18:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 06:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 06:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 03:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 01:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 00:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-19 ===
* 22:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 19:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 18:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 18:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 06:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 06:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 01:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-18 ===
* 22:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 22:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 19:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 12:31 dhinus: upgrade spicerack from 8.5 to 8.8 on cloudcumin*
* 10:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 07:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 07:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 03:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-17 ===
* 22:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 17:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 07:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 05:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 05:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 02:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 02:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-16 ===
* 23:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 23:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 17:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 11:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 11:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 09:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 09:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-07-15 ===
* 22:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:40 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 20:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 18:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy

=== 2024-07-11 ===
* 13:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 13:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 11:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 11:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 02:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'

=== 2024-07-08 ===
* 17:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T309789|T309789]])
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-07-06 ===
* 14:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-07-05 ===
* 10:33 arturo: aborrero@cloudcephmon1001:~$ sudo ceph osd unset norebalance
* 10:31 arturo: aborrero@cloudcephmon1001:~$ sudo ceph osd unset noin
* 08:56 arturo: installing nova/glance/cinder security updates [[phab:T369138|T369138]]

=== 2024-07-04 ===
* 20:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 20:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:52 dcaro: rebooting cloudcontrol1007 due to systemd-journal service failing to start
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-03 ===
* 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 16:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 12:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-02 ===
* 19:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-01 ===
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 14:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-06-27 ===
* 22:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet'
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1059.eqiad.wmnet'
* 19:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1059.eqiad.wmnet'
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1059.eqiad.wmnet'
* 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet'
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1064.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet'
* 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1067.eqiad.wmnet'
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet'
* 17:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1058.eqiad.wmnet'
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet'
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1066.eqiad.wmnet'
* 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet'
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1065.eqiad.wmnet'
* 17:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet'
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1057.eqiad.wmnet'
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 13:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 12:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet'
* 12:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirtlocal1001.eqiad.wmnet'
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 12:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 10:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 08:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 08:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 07:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 07:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2024-06-26 ===
* 22:32 andrewbogott: disabled all g3.* flavors in eqiad1
* 19:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 17:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 11:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 10:03 dcaro: taking cloudcephosd1006 out of the pool ([[phab:T348643|T348643]])
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 10:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-06-25 ===
* 22:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2003-dev.codfw.wmnet'
* 22:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2003-dev.codfw.wmnet'
* 22:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2002-dev.codfw.wmnet'
* 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2002-dev.codfw.wmnet'
* 22:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codfw.wmnet'
* 22:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2001-dev.codw.wmnet'
* 22:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codw.wmnet'
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2005-dev.codfw.wmnet'
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2004-dev.codfw.wmnet'
* 16:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt2006-dev.codfw.wmnet'
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2006-dev.codfw.wmnet'
* 03:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-24 ===
* 19:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet'
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056.eqiad.wmnet'
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet'
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet'
* 16:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet'
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet'

=== 2024-06-21 ===
* 10:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:35 wmbot~arturo@nostromo: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97)
* 10:35 wmbot~arturo@nostromo: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:35 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:35 wmbot~arturo@nostromo: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 09:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 08:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T368129|T368129]])
* 08:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T368129|T368129]])
* 04:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 4e612eb8-04e1-4541-941d-{{Gerrit|a05519eed60a}}
* 04:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 4e612eb8-04e1-4541-941d-{{Gerrit|a05519eed60a}}

=== 2024-06-20 ===
* 21:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 14789ac1-bc06-4677-9bb0-{{Gerrit|66c16c887427}}
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 14789ac1-bc06-4677-9bb0-{{Gerrit|66c16c887427}}
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet'
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet'
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet'
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1052.eqiad.wmnet'
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet'
* 13:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 614f9c99-86f1-410f-8ef5-{{Gerrit|e33d23215ff5}}
* 13:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 614f9c99-86f1-410f-8ef5-{{Gerrit|e33d23215ff5}}
* 13:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1051.eqiad.wmnet'
* 13:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet'
* 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1050.eqiad.wmnet'
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet'
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049.eqiad.wmnet'
* 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet'
* 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet'
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet'
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet'
* 10:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet'
* 09:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet'
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet'
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1045.eqiad.wmnet'
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 04:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server d0b1d9d5-1aec-4a05-a2d7-{{Gerrit|6d8522a365dc}}
* 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server d0b1d9d5-1aec-4a05-a2d7-{{Gerrit|6d8522a365dc}}
* 04:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 2d2e8925-b50f-483a-82ee-{{Gerrit|e6a1c588e5be}}
* 04:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 2d2e8925-b50f-483a-82ee-{{Gerrit|e6a1c588e5be}}
* 04:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 138be95d-93ad-4a85-9245-{{Gerrit|a0a508711555}}
* 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 138be95d-93ad-4a85-9245-{{Gerrit|a0a508711555}}
* 04:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server a3e945dc-3548-47fa-8ce3-{{Gerrit|bf1426ff3b15}}
* 04:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 0258b810-29af-448a-af5e-{{Gerrit|ed39e19286df}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server a3e945dc-3548-47fa-8ce3-{{Gerrit|bf1426ff3b15}}
* 04:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 37e23659-4516-4bd8-a9be-{{Gerrit|4cc55def5560}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 0258b810-29af-448a-af5e-{{Gerrit|ed39e19286df}}
* 04:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 7da824f3-c9f3-4460-b9b2-{{Gerrit|2a894268a7ca}}
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 37e23659-4516-4bd8-a9be-{{Gerrit|4cc55def5560}}
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 63b82d38-3026-408f-8dcc-{{Gerrit|0ecbd1a3c870}}
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 7da824f3-c9f3-4460-b9b2-{{Gerrit|2a894268a7ca}}
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 7cb371bb-a53a-4e65-a1cf-{{Gerrit|f1a8264a9166}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 63b82d38-3026-408f-8dcc-{{Gerrit|0ecbd1a3c870}}
* 04:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 6616fcbf-a49e-4e03-b735-{{Gerrit|84d31b535c08}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 6616fcbf-a49e-4e03-b735-{{Gerrit|84d31b535c08}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 7cb371bb-a53a-4e65-a1cf-{{Gerrit|f1a8264a9166}}
* 04:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 6f1171db-9d7d-466f-aaa7-{{Gerrit|cb14e1a6af41}}
* 04:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 6f1171db-9d7d-466f-aaa7-{{Gerrit|cb14e1a6af41}}
* 04:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 1dc3fcee-cf27-4351-ad60-{{Gerrit|384478624ea3}}
* 04:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 1dc3fcee-cf27-4351-ad60-{{Gerrit|384478624ea3}}
* 04:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 3df62375-e75d-4068-9c71-{{Gerrit|13519b6bf927}}
* 04:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 3af9b40f-d29d-4216-9b64-{{Gerrit|0ebb10f94c7c}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 3df62375-e75d-4068-9c71-{{Gerrit|13519b6bf927}}
* 04:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server ae9fa949-e4ce-4ffb-ad9f-{{Gerrit|4e5a3812d031}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server ae9fa949-e4ce-4ffb-ad9f-{{Gerrit|4e5a3812d031}}
* 04:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 452dc8d3-6ee0-412c-90ba-{{Gerrit|66f21f9d09c1}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 3af9b40f-d29d-4216-9b64-{{Gerrit|0ebb10f94c7c}}
* 04:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 452dc8d3-6ee0-412c-90ba-{{Gerrit|66f21f9d09c1}}
* 04:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 47b0da1d-e50a-42c1-8cd9-{{Gerrit|dc255cc2f1a3}}
* 04:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 47b0da1d-e50a-42c1-8cd9-{{Gerrit|dc255cc2f1a3}}
* 02:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T368007|T368007]])
* 02:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T368007|T368007]])
* 02:05 andrewbogott: cloudvirt1063 is unresponsive, cycling power from racadm

=== 2024-06-19 ===
* 15:43 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1044.eqiad.wmnet'
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 15:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet'
* 12:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 12:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet'
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet'
* 11:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet'
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1042.eqiad.wmnet'
* 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server f667d3c2-379a-48d7-ad44-{{Gerrit|4f3933bdb871}}
* 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server f667d3c2-379a-48d7-ad44-{{Gerrit|4f3933bdb871}}
* 01:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 71e4296d-039d-4452-9d92-{{Gerrit|69b9f8eb3aba}}
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 71e4296d-039d-4452-9d92-{{Gerrit|69b9f8eb3aba}}
* 01:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server a7725cd2-6162-41a3-8add-{{Gerrit|4dc0668b233b}}
* 01:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server a7725cd2-6162-41a3-8add-{{Gerrit|4dc0668b233b}}
* 01:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}

=== 2024-06-18 ===
* 21:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2024-06-17 ===
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T364457|T364457]])
* 20:03 andrewbogott: repaced ovs hosts in the 'ceph' aggregate
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T364457|T364457]])
* 18:16 andrewbogott: temporarily removing all ovs hosts from the 'ceph' aggregate so the scheduler will stop putting linuxbridge hosts on ovs hosts and breaking them
* 17:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T364457|T364457]])
* 13:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet'

=== 2024-06-14 ===
* 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet'
* 13:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet'

=== 2024-06-13 ===
* 16:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet'
* 13:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet'
* 13:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2003-dev.codfw.wmnet'
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2003-dev.codfw.wmnet'

=== 2024-06-12 ===
* 11:49 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1031.eqiad.wmnet'
* 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet'
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet'
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet'
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet'
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet'

=== 2024-06-11 ===
* 15:55 dcaro: restarting mon service on cloudcephmon1002 to try to release the 2 stuck ops left
* 13:30 taavi: pin all existing eqiad1 flavors to linuxbridge hypervisors [[phab:T364458|T364458]]
* 12:28 taavi: add all existing eqiad1 cloudvirts to new network-linuxbridge aggregate [[phab:T364458|T364458]]
* 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-07 ===
* 19:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-05 ===
* 11:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-06-04 ===
* 15:49 taavi: drop hopefully-unused 68.10.in-addr.arpa. from designate [[phab:T361220|T361220]]

=== 2024-05-30 ===
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-29 ===
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T364984|T364984]])
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T364984|T364984]])
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T364984|T364984]])
* 18:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T364984|T364984]])

=== 2024-05-28 ===
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-21 ===
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 08:18 taavi: stop neutron services on cloudnet1005 [[phab:T364459|T364459]]

=== 2024-05-20 ===
* 14:46 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:46 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:20 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:19 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:18 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:18 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:17 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:17 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:16 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:16 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:15 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:15 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs

=== 2024-05-16 ===
* 08:55 taavi: delete 'monitoring' project https://phabricator.wikimedia.org/T365105

=== 2024-05-15 ===
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T319184|T319184]])
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-05-14 ===
* 19:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-10 ===
* 13:23 andrewbogott: deploying updated 2024.1 Horizon

=== 2024-05-09 ===
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-25 ===
* 21:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T356287|T356287]])
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T356287|T356287]])
* 21:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1033' ([[phab:T356287|T356287]])
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033' ([[phab:T356287|T356287]])
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T356287|T356287]])
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T356287|T356287]])

=== 2024-04-23 ===
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2002-dev.codfw.wmnet'
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2002-dev.codfw.wmnet'

=== 2024-04-18 ===
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2008-dev.codfw.wmnet'
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2008-dev.codfw.wmnet'

=== 2024-04-17 ===
* 14:12 dcaro: deleting dns leaks
* 01:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])

=== 2024-04-16 ===
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 20:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2001-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2001-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T356287|T356287]])

=== 2024-04-15 ===
* 11:19 taavi: update spicerack to 8.5.0 on cloudcumin2001

=== 2024-04-10 ===
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-09 ===
* 18:13 andrewbogott: rebooting cloudinfra-cloudvps-puppetserver-1; unresponsive
* 13:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-05 ===
* 14:37 taavi: run maintain-replica-indexes on all web replicas [[phab:T361945|T361945]]
* 14:27 taavi: run maintain-replica-indexes on remaining analytics replicas [[phab:T361945|T361945]]
* 14:17 taavi: run maintain-replica-indexes on clouddb1017 [[phab:T361945|T361945]]

=== 2024-04-04 ===
* 18:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-03 ===
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T319184|T319184]])
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T319184|T319184]])
* 10:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T319184|T319184]])
* 10:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T319184|T319184]])
* 09:17 taavi: manually delete prometheus-node-textfile-wmcs-dnsleaks.service and related files from cloudservices1005/6, leftovers of the designate api to cloudcontrol migration
* 09:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T319184|T319184]])
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2024-04-02 ===
* 15:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T319184|T319184]])
* 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T319184|T319184]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 12:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 12:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-03-27 ===
* 11:47 taavi: deleting about 2k stale puppet certs by running wmcs-puppetcertleaks in delete mode

=== 2024-03-22 ===
* 10:25 dcaro: adding back cloudcephosd1034 to the pool after doing the performance tests ([[phab:T348643|T348643]])

=== 2024-03-21 ===
* 22:07 andrewbogott: doing dist-upgrade on cloudcontrol nodes to get mariadb upgraded for [[phab:T357133|T357133]]
* 22:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T357133|T357133]])
* 22:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T357133|T357133]])
* 11:08 dcaro: restarting nova-api on cloudcontrol1007

=== 2024-03-20 ===
* 17:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:27 dcaro: turning off cloudcephosd1030 to swap some disks ([[phab:T348643|T348643]])
* 15:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T348643|T348643]])
* 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2024-03-19 ===
* 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 13:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2024-03-18 ===
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-03-09 ===
* 16:43 andrewbogott: restarted nova-api on cloudcontrol1006

=== 2024-03-08 ===
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:21 arturo: restarted nova-api in cloudcontrol1007, it was complaining about mysql broken pipe
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 10:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 10:35 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2024-03-07 ===
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance

=== 2024-03-06 ===
* 17:46 dhinus: running &quot;wmcs-dnsleaks --delete&quot; to clean up 2 leaked records (tools-sgeweblight-10-32)
* 15:36 dcaro: renewing puppet ca cert for cloud-puppetmaster-03
* 15:24 dcaro: renewing puppet ca cert for cloudinfra-internal puppetmaster

=== 2024-03-04 ===
* 14:56 dhinus: delete project &quot;loggerdiscordbot&quot; in favor of new project &quot;discordbots&quot; [[phab:T358337|T358337]],[[phab:T358427|T358427]]
* 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reboot_node (exit_code=0) ([[phab:T359049|T359049]])
* 12:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reboot_node ([[phab:T359049|T359049]])

=== 2024-03-01 ===
* 14:59 taavi: removing wmf-auto-restart-cron from all VMs without cron via cumin - https://gerrit.wikimedia.org/r/c/operations/puppet/+/1007328/ [[phab:T358343|T358343]]
* 12:07 dcaro: restarted nova-api on cloudcontrol100* as it was very slow
* 12:04 dcaro: restarted nova-api on cloudcontrol1005 as it was very slow

=== 2024-02-27 ===
* 18:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-26 ===
* 10:02 arturo: deleting nskaggs account from gerrit's wmcs-trusted group

=== 2024-02-22 ===
* 13:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-21 ===
* 13:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) ([[phab:T319184|T319184]])
* 13:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance ([[phab:T319184|T319184]])
* 13:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T319184|T319184]])
* 13:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T319184|T319184]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-02-20 ===
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:45 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=0) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])

=== 2024-02-19 ===
* 12:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 10:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=0) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=99) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])

=== 2024-02-15 ===
* 17:34 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.roll_reboot_cloudnets ([[phab:T356975|T356975]])
* 15:34 taavi: restart radosgw in eqiad as I am seeing 500 errors
* 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:58 dhinus: restore correct aggregate &quot;localdisk&quot; for cloudvirtlocal1001 and remove &quot;maintenance&quot;
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T319184|T319184]])
* 05:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet[B&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet[B&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2024-02-14 ===
* 22:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtXXXX.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtXXXX.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:49 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:49 wmbot~andrew@bullseye: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97)
* 19:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:48 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:45 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:45 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:38 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:38 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:36 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:36 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:34 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:34 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:32 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 17:13 wmbot~fran@wmf3169: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtlocal1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T356975|T356975]])
* 17:12 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtlocal1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T356975|T356975]])
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:09 taavi: creating some missing $PROJECT.wmcloud.org. DNS zones
* 14:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:21 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 09:31 taavi: failover all dumps traffic to clouddumps1001
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 08:16 taavi: reboot clouddumps1001 for kernel updates

=== 2024-02-13 ===
* 17:07 wmbot~taavi@runko: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:07 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:06 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:06 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::eqiad1::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::eqiad1::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P:openstack::eqiad1::nova::compute::service'
* 16:02 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P:openstack::eqiad1::nova::compute::service'
* 16:01 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:01 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:10 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) ([[phab:T356975|T356975]])
* 14:55 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T356975|T356975]])

=== 2024-02-12 ===
* 17:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0)
* 17:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0)
* 17:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:23 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99)
* 17:21 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 17:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 17:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:36 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 16:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 15:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 15:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99)
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 01:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:34 andrewbogott: resetting eqiad1 rabbitmq in hopes of resolving neutron double message warnings
* 01:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-11 ===
* 21:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:57 andrewbogott: running wmcs.openstack.restart_openstack for all eqiad1 services
* 20:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 20:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-08 ===
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:43 taavi: deploy change to exclude cloud-private networks from general egress NAT https://phabricator.wikimedia.org/T356850

=== 2024-02-06 ===
* 20:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:09 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 02:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-05 ===
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:55 andrewbogott: rebuilt bookworm base image in eqiad1 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/992677

=== 2024-02-02 ===
* 13:54 arturo: [codfw1dev] cleanup /etc/network/interfaces on cloudlb2003-dev from puppet leftovers

=== 2024-02-01 ===
* 10:55 taavi: invite aborrero to /repos/cloud, /toolforge-repos, /cloudvps-repos on gitlab

=== 2024-01-29 ===
* 12:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-26 ===
* 09:01 taavi: joining cloudrabbit1001/2 to the cluster on 1003 [[phab:T345610|T345610]]

=== 2024-01-25 ===
* 16:48 andrewbogott: taavi just moved all rabbitmq traffic to cloudrabbit1003 as part of [[phab:T345610|T345610]]
* 16:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:27 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:27 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-2.tools.eqiad1.wikimedia.cloud to the cluster
* 13:15 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:15 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-1.tools.eqiad1.wikimedia.cloud to the cluster
* 12:48 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 12:48 wmbot~taavi@runko: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-1.toolsbeta.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-24 ===
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-22 ===
* 16:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:56 andrewbogott: restarting openstack sevices on eqiad1 to clean up from the mariadb restarts
* 15:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:07 andrewbogott: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/992192 and resetting galera cluster in eqiad1

=== 2024-01-21 ===
* 05:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 05:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-19 ===
* 23:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-18 ===
* 20:10 taavi: mysql:labsdbaccounts@m5-master.eqiad.wmnet [labsdbaccounts]&gt; update account_host set status = 'absent' where id = 137613;
* 12:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 12:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-101.tools.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-17 ===
* 15:17 andrewbogott: &quot;systemctl restart mariadb@s4.service mariadb@s6.service&quot; on clouddb1015. System is in danger of oom and there are no obvious long queries running

=== 2024-01-16 ===
* 14:25 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:25 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 13:55 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:55 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:54 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:54 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:53 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 13:53 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:52 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:52 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:35 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 11:34 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 11:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 09:41 taavi: drop dbproxy1018/9 grants from all clouddb hosts [[phab:T346947|T346947]]
* 09:24 taavi: move cloudvirt2004-dev from 'failed' to 'active' in netbox - seems like that was for [[phab:T348531|T348531]] which is now resolved

=== 2024-01-15 ===
* 14:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 14:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 14:32 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 12:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-11 ===
* 10:51 dcaro: restarting striker.service on cloudweb1003 as it seems non-responsive

=== 2024-01-10 ===
* 17:43 bd808: Blocking Developer accounts connected to invalid/legacy wikimedia.org email addresses ([[phab:T218239|T218239]])
* 16:48 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0) ([[phab:T346631|T346631]])
* 16:48 wmbot~fran@wmf3169: test message3 from local cookbook ([[phab:T346631|T346631]])
* 16:48 wmbot~fran@wmf3169: START - Cookbook wmcs.do_log_msg ([[phab:T346631|T346631]])

=== 2024-01-09 ===
* 17:50 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0) ([[phab:T346631|T346631]])
* 17:49 wmbot~fran@wmf3169: test message2 from local cookbook ([[phab:T346631|T346631]])
* 17:49 wmbot~fran@wmf3169: START - Cookbook wmcs.do_log_msg ([[phab:T346631|T346631]])
* 17:43 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])
* 17:43 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])
* 17:42 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])

=== 2024-01-08 ===
* 15:52 taavi: verify wmcloud.org, wmflabs.org and toolforge.org in gmail postmaster console to figure out how much google likes us ([[phab:T354112|T354112]])

=== 2024-01-07 ===
* 19:34 andrewbogott: removed cloudvirt1063 from 'ceph' aggregate, added to 'maintenance' aggregate   [[phab:T353408|T353408]]
* 19:34 andrewbogott: evacuating all VMs from cloudvirt1063.  [[phab:T353408|T353408]]

=== 2024-01-02 ===
* 16:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T353408|T353408]])
* 16:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T353408|T353408]])
* 10:22 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:22 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-12-31 ===
* 21:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:35 andrewbogott: running openstack service restart cookbook in eqiad1 in response to a bunch of service down alerts
* 21:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-21 ===
* 16:52 dhinus: puppet node deactivate cloudvirt1063.eqiad.wmnet [[phab:T353406|T353406]]
* 03:01 andrewbogott: restarting mariadb on cloudcontrol1005, hoping to get Galera back in sync

=== 2023-12-20 ===
* 19:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-18 ===
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:23 andrewbogott: restarting all eqiad1 openstack services after a rabbitmq upgrade/rebuild for [[phab:T353646|T353646]]
* 15:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-15 ===
* 13:13 dcaro: restarted nova-fullstack on codfw as it was stuck (and alerting through stale prometheus file)

=== 2023-12-14 ===
* 00:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 00:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 00:25 andrewbogott: evacuating hosts from cloudvirt1063 and depooling. [[phab:T353406|T353406]]

=== 2023-12-13 ===
* 16:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=99)

=== 2023-12-12 ===
* 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 17:45 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-100.tools.eqiad1.wikimedia.cloud to the cluster
* 16:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:11 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-99.tools.eqiad1.wikimedia.cloud to the cluster
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:49 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-98.tools.eqiad1.wikimedia.cloud to the cluster

=== 2023-12-10 ===
* 18:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:27 andrewbogott: restarting all openstack API servers, hoping to make things a bit more responsive
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-08 ===
* 12:00 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99)  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud ([[phab:T353055|T353055]])
* 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud ([[phab:T353055|T353055]])
* 11:58 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=97)  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud
* 11:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud
* 09:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) ([[phab:T345084|T345084]])
* 09:32 dcaro: restarting nova and keystone as they are getting too slow ([[phab:T345084|T345084]])
* 09:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack ([[phab:T345084|T345084]])
* 09:32 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) ([[phab:T345084|T345084]])
* 09:31 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack ([[phab:T345084|T345084]])

=== 2023-12-07 ===
* 13:12 dcaro: rebooting cloudcephosd1001 to make sure puppet7 migration went ok

=== 2023-12-04 ===
* 00:08 andrewbogott: rebooting cloudcontrol1006 to recover from full disk error

=== 2023-12-03 ===
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-02 ===
* 12:27 taavi: powercycle cloudvirt1063 [[phab:T352595|T352595]]
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:28 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-97.tools.eqiad1.wikimedia.cloud to the cluster
* 11:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-96.tools.eqiad1.wikimedia.cloud to the cluster
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 10:50 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-95.tools.eqiad1.wikimedia.cloud to the cluster
* 00:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:21 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-94.tools.eqiad1.wikimedia.cloud to the cluster
* 00:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-93.tools.eqiad1.wikimedia.cloud to the cluster
* 00:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-92.tools.eqiad1.wikimedia.cloud to the cluster
* 00:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-91.tools.eqiad1.wikimedia.cloud to the cluster
* 00:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:05 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-90.tools.eqiad1.wikimedia.cloud to the cluster
* 00:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:01 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-89.tools.eqiad1.wikimedia.cloud to the cluster

=== 2023-12-01 ===
* 17:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:19 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T351171|T351171]])
* 16:19 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 15:49 andrewbogott: reimaging cloudcontrol1005 due to widespread misbehavior
* 14:24 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T351171|T351171]])
* 14:20 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 14:19 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97) ([[phab:T351171|T351171]])
* 14:18 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 13:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:55 taavi: restart neutron-rpc-server.service on eqiad1 cloudcontrols

=== 2023-11-30 ===
* 20:44 andrewbogott: generating to application credentials for the tests that run on tf-infra-test
* 19:54 andrewbogott: reimaged cloudrabbit100[23] after https://gerrit.wikimedia.org/r/c/operations/puppet/+/979127.  I didn't reimage 1001 because that will require rebuilding the whole cluster but I did remove the related packages.
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet'
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet'
* 17:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet'
* 17:47 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet'
* 17:42 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:30 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:19 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:19 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:09 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:59 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:54 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:39 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:34 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:15 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:10 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:05 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:00 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:31 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:22 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:17 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:17 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:13 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:07 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:38 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:16 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:03 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 13:55 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 13:43 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T348843|T348843]])
* 13:43 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T348843|T348843]])
* 12:18 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 12:04 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:59 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:45 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:44 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:28 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T348843|T348843]])

=== 2023-11-29 ===
* 15:27 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:15 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:59 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T348843|T348843]])
* 14:50 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T348843|T348843]])

=== 2023-11-28 ===
* 14:18 taavi: moving wiki replica DNS to use cloudlbs instead of the old proxy VMs [[phab:T346947|T346947]]

=== 2023-11-27 ===
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-24 ===
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-22 ===
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 13:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-21 ===
* 10:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-20 ===
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-17 ===
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-16 ===
* 12:09 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:05 taavi@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 11:23 dhinus: upgraded spicerack from 8.0.2 to 8.0.3 on cloudcumins
* 05:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 05:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-15 ===
* 09:50 taavi: move cloudlb hosts to use the nftables firewall backend

=== 2023-11-14 ===
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 12:03 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T345811|T345811]])
* 11:44 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T345811|T345811]])
* 10:10 taavi: restart kiwix-mirror-update on clouddumps1001
* 05:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 05:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 04:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 04:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 04:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 04:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 03:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 03:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 03:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 03:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])

=== 2023-11-13 ===
* 22:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 22:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 22:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 22:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 21:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 19:31 andrewbogott: rebooting cloudcontrol2005-dev, trying to fix general misbehavior
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 19:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1033.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1032.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet'
* 17:09 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T345811|T345811]])
* 17:09 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T345811|T345811]])
* 16:57 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T345811|T345811]])
* 15:37 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T345811|T345811]])
* 15:19 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T345811|T345811]])
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-11 ===
* 02:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 00:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1058'
* 00:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1058'
* 00:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 00:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot

=== 2023-11-09 ===
* 21:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:54 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1025.eqiad.wmnet' ([[phab:T345811|T345811]])
* 14:52 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1025.eqiad.wmnet' ([[phab:T345811|T345811]])
* 14:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 14:50 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])

=== 2023-11-08 ===
* 16:43 andrewbogott: created foundationmemory project for [[phab:T350760|T350760]]

=== 2023-11-06 ===
* 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 15:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)

=== 2023-11-05 ===
* 00:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-11-04 ===
* 23:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 22:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 04:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)

=== 2023-11-03 ===
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:13 dhinus: triggering neutron failover from cloudnet1005 to cloudnet1006 ([[phab:T345811|T345811]])
* 06:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 01:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 01:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)

=== 2023-11-02 ===
* 20:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:32 taavi: merged cloudcontrol firewall cleanup patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/971211
* 16:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 16:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 13:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 11:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 11:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 07:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 07:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 05:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 03:47 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 03:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)

=== 2023-11-01 ===
* 23:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 21:29 taavi: re-enable puppet on cloudcontrol2006-dev.codfw.wmnet which has fallen off of puppetdb
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 20:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 19:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 14:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 09:04 taavi: reset local cookbook changes on cloudcumin1001 which were causing issues with puppet runs
* 09:02 taavi: restart nova-fullstack which had had some issues after yesterday's cloudcontrol1007 reimage
* 02:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 00:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-10-31 ===
* 23:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 18:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 14:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 10:16 dhinus: upgrading mariadb-server in cloudcontrol1005 ([[phab:T345811|T345811]])
* 10:04 dhinus: upgrading mariadb-server in cloudcontrol1006 ([[phab:T345811|T345811]])
* 09:51 dhinus: upgrading mariadb-server in cloudcontrol1007, second attempt ([[phab:T345811|T345811]])
* 06:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 05:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 02:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 02:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 02:27 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 02:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-10-30 ===
* 17:40 andrewbogott: rebooting tools-db-1.tools.eqiad1.wikimedia.cloud for yet another oom death
* 17:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 16:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 dhinus: upgrading mariadb-server in cloudcontrol1007 ([[phab:T345811|T345811]])
* 16:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-28 ===
* 07:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)

=== 2023-10-27 ===
* 15:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 15:38 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 15:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 15:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 15:29 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 15:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 15:27 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:09 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:05 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 09:04 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-26 ===
* 14:02 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 09:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-25 ===
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
* 11:00 taavi: update cloudcumins to spicerack 8.x
* 10:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster

=== 2023-10-24 ===
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-10-23 ===
* 16:48 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:06 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project catalyst in eqiad1
* 15:06 wm-bot2: fran@wmf3169 START - Cookbook wmcs.vps.create_project for project catalyst in eqiad1
* 10:36 taavi: merged change https://gerrit.wikimedia.org/r/c/operations/puppet/+/966494 which touches the pdns web server config

=== 2023-10-20 ===
* 15:17 dcaro: upgraded cloudcephosd1004 to v15 ([[phab:T349363|T349363]])
* 14:25 dcaro: upgraded cloudcephosd1003 to v15 ([[phab:T349363|T349363]])
* 13:20 dcaro: upgraded cloudcephosd1002 to v15 ([[phab:T349363|T349363]])
* 10:33 dcaro: upgraded cloudcephosd1001 to v15 ([[phab:T349363|T349363]])
* 08:26 dcaro: codfw ceph enabled diskprediction_local module, will take a bit to populate/start getting predictions ([[phab:T348716|T348716]])

=== 2023-10-17 ===
* 15:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T349109|T349109]])
* 15:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T349109|T349109]])

=== 2023-10-16 ===
* 03:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-10-13 ===
* 23:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 08:31 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 08:30 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 08:20 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])

=== 2023-10-12 ===
* 17:16 wm-bot2: fran@wmf3169 END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) ([[phab:T341285|T341285]])
* 17:16 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])

=== 2023-10-11 ===
* 10:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 10:36 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 07:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 06:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-10 ===
* 17:25 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:50 dcaro: removing ~100 dangling backup snapshots from eqiad1-compute ceph pool
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 12:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 11:38 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 11:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 11:33 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 11:32 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 11:00 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) ([[phab:T341285|T341285]])
* 11:00 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 10:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 09:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 09:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 09:43 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 08:19 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:18 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:18 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:17 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:17 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-09 ===
* 17:16 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:26 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:18 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:49 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:26 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 13:55 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 13:45 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 13:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:04 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:10 dcaro: undrained cephosd1011
* 09:09 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 09:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:06 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 09:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:05 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 09:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 07:35 taavi: restart postgresql on cloudbackup2001 [[phab:T348431|T348431]]

=== 2023-10-05 ===
* 16:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:29 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 16:24 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:11 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:30 arturo: operating on cloudgw @ eqiad1 ([[phab:T347469|T347469]])
* 14:07 wm-bot2: dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 12:55 arturo: doing cloudgw maintenance operations [[phab:T347469|T347469]]
* 11:54 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_rack
* 10:57 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 09:54 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 09:52 arturo: [codfw1dev] aborrero@cloudcontrol2001-dev:~ $ sudo keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
* 09:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 09:40 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 09:33 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 08:50 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 00:04 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)

=== 2023-10-04 ===
* 20:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 15:55 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 14:54 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 14:44 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 14:41 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 14:40 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 13:50 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:08 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:21 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:12 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-03 ===
* 19:38 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:35 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:32 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 11:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 08:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:43 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 08:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:39 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 08:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:23 dcaro: set .rgw.root pool on eqiad as rgw app (`ceph osd pool application enable .rgw.root rgw`)

=== 2023-10-02 ===
* 17:39 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:30 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:58 taavi: cloudcontrol1005,7: `sudo systemctl reset-failed keystone_sync_keys_from_cloudcontrol1006.eqiad.wmnet.service`
* 13:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:40 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:37 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 12:26 arturo: [codfw1dev] run `update domains set master = '185.15.57.25:5354 185.15.57.26:5354 172.20.5.9:5354 172.20.5.8:5354';` in cloudservies2005-dev
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])
* 11:55 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])
* 11:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 08:44 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])

=== 2023-09-29 ===
* 12:43 taavi: taavi@cloudcontrol1005 ~ $ os subnet set a69bdfad-d7d2-4cfa-8231-{{Gerrit|3d6d3e0074c9}} --no-dns-nameservers --dns-nameserver 172.20.255.1
* 08:36 taavi: start script to fix networking on broken bullseye instances [[phab:T347665|T347665]]

=== 2023-09-28 ===
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 12:01 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:48 arturo: rebooting cloudgw1001/1002 for sysctl and kernel upgrades

=== 2023-09-27 ===
* 12:07 arturo: merging cloudgw firewall changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/961360
* 09:36 taavi: move maintain-dbusers to cloudcontrol1005
* 01:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-26 ===
* 17:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-24 ===
* 15:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-23 ===
* 08:40 taavi: restart keystone

=== 2023-09-22 ===
* 14:03 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 14:02 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 14:02 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 14:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 14:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.undrain_node (exit_code=0)
* 14:00 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 14:00 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:56 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.undrain_node (exit_code=0)
* 13:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.undrain_node (exit_code=99)
* 13:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:48 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.undrain_node (exit_code=99)
* 13:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 13:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:46 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:44 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:41 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:37 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:33 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:33 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 11:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node

=== 2023-09-21 ===
* 16:35 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0)  on tools-db-3.tools.eqiad1.wikimedia.cloud
* 16:34 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs  on tools-db-3.tools.eqiad1.wikimedia.cloud
* 02:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-20 ===
* 21:49 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:45 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:38 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 21:35 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:31 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:12 arturo: moving openstack API endpoint to cloudlb ([[phab:T346439|T346439]])
* 10:28 arturo: running SQL command `update domains set master=&quot;172.20.1.5:5354 172.20.2.4:5354 185.15.56.162:5354 185.15.56.163:5354&quot;;` on cloudservices1005/1006 ([[phab:T346042|T346042]])
* 10:06 arturo: running SQL command `update domains set master=&quot;185.15.56.162:5354 185.15.56.163:5354&quot;` on cloudservices1005/1005 ([[phab:T346042|T346042]])

=== 2023-09-19 ===
* 18:54 andrewbogott: depooling clouddb1019 to let it recover from high memory use.  [[phab:T346826|T346826]]
* 15:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-18 ===
* 11:53 taavi: update designate urls in Keystone to point to openstack-next, until cloudlb is serving the main openstack address [[phab:T346042|T346042]]
* 11:40 arturo: decomission cloudservices1005 [[phab:T346042|T346042]] in preparation for re-racking
* 08:45 arturo: hardcode `185.15.56.161 openstack.eqiad1.wikimediacloud.org` in /etc/hosts in cloudcontrol1005 for [[phab:T346441|T346441]]

=== 2023-09-15 ===
* 11:43 arturo: merging NAT change for [[phab:T346426|T346426]] in cloudgw
* 10:33 arturo: faiolver cloudgw1001 into cloudgw1002, investigating a nftables syntax error ([[phab:T346432|T346432]])

=== 2023-09-14 ===
* 21:25 wm-bot2: andrew@bullseye END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:25 wm-bot2: andrew@bullseye START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:23 wm-bot2: andrew@bullseye END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:23 wm-bot2: andrew@bullseye START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 17:13 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 17:06 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 17:01 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:51 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:29 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:29 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:25 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) ([[phab:T345810|T345810]])
* 16:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:12 topranks: DNS operation: remove old DNS entry for ns0.openstack.eqiad1.wikimediacloud.org. on wikimedia authdns (was pointing to 208.80.154.148)
* 16:10 topranks: DNS operation: add new DNS entry for ns0.openstack.eqiad1.wikimediacloud.org. on wikimedia authdns pointing to 185.15.56.162
* 14:42 arturo: DNS operation: route 208.80.154.148 to cloudservices1006 in anticipation of cloudservices1005 decom ([[phab:T346042|T346042]])
* 12:11 arturo: enable puppet on cloudservices1006 to drop local NAT hacks and enable new DNS auth IP address ([[phab:T346042|T346042]])

=== 2023-09-13 ===
* 17:11 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) ([[phab:T345811|T345811]])
* 17:08 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 17:04 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 17:04 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:57 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:53 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:53 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:49 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:49 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 01:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 01:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 01:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-12 ===
* 18:06 andrewbogott: update domains set master='185.15.56.163:5354 208.80.154.11:5354 10.64.151.4:5354'; on cloudservices1005 + cloudservices1006
* 17:59 andrewbogott: mysql:root@localhost [pdns]&gt; update domains set master='185.15.56.163:5354 208.80.154.11:5354';
* 17:59 andrewbogott: &quot;designate-manage pool update' on cloudservices1005 to remove cloudservices1004 from the pool
* 17:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-11 ===
* 12:36 arturo: update DNS resolver cloud-wide to use 172.20.255.1 ([[phab:T342621|T342621]])

=== 2023-09-10 ===
* 02:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 02:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-07 ===
* 10:09 dhinus: reimaging cloudcontrol2001-dev to bookworm ([[phab:T345810|T345810]])

=== 2023-09-06 ===
* 16:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97)
* 16:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:53 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:37 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:37 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:37 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:35 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:35 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:34 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:34 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:23 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97)
* 16:23 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:13 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99)
* 16:12 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance

=== 2023-09-05 ===
* 12:34 arturo: synced pdns database from cloudservices1004 to cloudservices1006 ([[phab:T345240|T345240]])
* 12:18 arturo: updating pools.yaml in all cloudservices designate nodes ([[phab:T345240|T345240]])
* 10:54 arturo: running SQL command `update domains set master=&quot;208.80.154.11:5354 208.80.154.148:5354 10.64.151.4:5354&quot;;` on all 3 cloudservices nodes ([[phab:T345240|T345240]])

=== 2023-09-04 ===
* 15:58 arturo: stop and mask designate-sink.service @ cloudservices1006
* 14:19 arturo: started all designate services on cloudservices1006 [[phab:T345240|T345240]]
* 10:46 arturo: added designate galera DB grants for cloudlb [[phab:T345240|T345240]]
* 08:40 arturo: stopped all designate services on cloudservices1006 [[phab:T345240|T345240]]

=== 2023-09-01 ===
* 10:28 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.network.tests (exit_code=1) ([[phab:T345282|T345282]])
* 10:25 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.network.tests ([[phab:T345282|T345282]])

=== 2023-08-31 ===
* 15:14 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=99)
* 15:14 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 12:49 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudnet.show (exit_code=0)
* 12:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudnet.show
* 12:48 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 12:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 12:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 12:46 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:46 wm-bot2: fran@wmf3169 START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2023-08-30 ===
* 13:07 wm-bot2: dcaro testing stuff

=== 2023-08-28 ===
* 15:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:05 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-23 ===
* 16:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:10 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:08 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:08 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-15 ===
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:32 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 19:32 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:01 andrewbogott: rebooting cloudvirt2001-dev in an attempt to figure out what's happening with bastions
* 15:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:42 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:39 dcaro: removed some logs from the cloudmetrics1003:/var/log/carbon/ directory and stopped the carbon processes (they were crashing and filling up the disk with logs)

=== 2023-08-14 ===
* 22:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:11 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:08 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-07 ===
* 09:39 taavi: cloud vps graphite service was disabled: https://wikitech.wikimedia.org/wiki/News/2023_Cloud_VPS_metrics_changes [[phab:T326266|T326266]]

=== 2023-08-03 ===
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0)
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.do_log_msg
* 13:07 wm-bot2: Test SAL log ([[phab:T341793|T341793]]) - cookbook ran by root@cloudcumin1001

=== 2023-07-31 ===
* 16:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0)
* 16:16 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 16:15 fnegri@cloudcumin1001: START - Cookbook wmcs.do_log_msg
* 15:33 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:42 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:35 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 13:55 andrewbogott: recreating the codfw1dev galera cluster according to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Galera -- mariadb is stopped (and won't start) on all three cloudcontrol nodes
* 13:39 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001

=== 2023-07-27 ===
* 10:52 arturo: adding cloud-private subnet to cloudnet1005/1006 hosts in eqiad1 ([[phab:T342619|T342619]])

=== 2023-07-19 ===
* 14:02 wm-bot2: Draining cloudvirt2002-dev.codfw.wmnet ([[phab:T335840|T335840]]) - cookbook ran by raymond@ubuntu
* 14:02 wm-bot2: Safe rebooting cloudvirt2002-dev.codfw.wmnet ([[phab:T335840|T335840]]) - cookbook ran by raymond@ubuntu

=== 2023-07-17 ===
* 15:55 arturo: cloudcontrol1005 was shutdown earlier today ([[phab:T341495|T341495]])
* 15:35 arturo: [codfw1dev] cloudweb2002-dev up and running after reracking ([[phab:T327919|T327919]])
* 15:11 arturo: [codfw1dev] powered off cloudweb2002-dev for reracking ([[phab:T327919|T327919]])
* 12:45 taavi: removing diamond from remaining buster instances [[phab:T317032|T317032]]

=== 2023-07-13 ===
* 10:48 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:38 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:33 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay

=== 2023-07-07 ===
* 10:28 taavi: backfilling &lt;nowiki&gt;{&lt;/nowiki&gt;project&lt;nowiki&gt;}&lt;/nowiki&gt;.wmcloud.org and other currently-named DNS zones to projects that don't have them

=== 2023-07-06 ===
* 17:02 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:00 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:00 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-07-04 ===
* 14:08 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:07 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001

=== 2023-06-30 ===
* 21:37 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-06-29 ===
* 19:49 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:47 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-06-26 ===
* 08:46 arturo: [codfw1dev] manually start mariadb service @ cloudinfra-db-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud

=== 2023-06-24 ===
* 19:21 wm-bot2: Created new flavor: g3.cores16.ram34.disk20 (id:7dd33202-32c3-4bc7-b2d4-{{Gerrit|10c2ebe7e5c5}}) - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Created new flavor: g3.cores16.ram34816.disk20.admin (id:d76925a8-b58b-489e-a3d9-{{Gerrit|c68c7744b551}}) - cookbook ran by andrew@bullseye

=== 2023-06-23 ===
* 16:45 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 13:59 andrewbogott: rebooting every VM in codfw1dev

=== 2023-06-13 ===
* 15:35 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 14:58 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye

=== 2023-06-12 ===
* 19:37 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 11:57 arturo: [codfw1dev] refresh various occurrences of old FQDNs in instance puppet via horizon ([[phab:T324992|T324992]])

=== 2023-06-08 ===
* 16:56 andrewbogott: deleting all Stretch base images from glance
* 16:45 andrewbogott: updated the bullseye image with https://cloud.debian.org/images/cloud/bullseye/20230601-1398/debian-11-genericcloud-amd64-20230601-1398.tar.xz
* 12:17 wm-bot2: Drained cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:17 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 02920314-1efe-4934-ad81-{{Gerrit|d2a6cf2e17ab}}, use this to unset) ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:16 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:06 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 769349bf-465f-4f0c-a8f3-{{Gerrit|f2423631ba7e}}, use this to unset) ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus

=== 2023-06-07 ===
* 10:06 dcaro: upgraded ruby2.5 to latest fixed version on all buster VMs
* 08:10 dcaro: downgrading ruby2.5 to previous backport on all buster VMs

=== 2023-06-06 ===
* 19:09 andrewbogott: also increased RAM and secgroup-rule quota for Trove  [[phab:T337882|T337882]]
* 19:06 andrewbogott: increased trove secgroups, instances, volumes quotas from 40 to 100.  Trove is too popular!  [[phab:T337882|T337882]]
* 18:31 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 18:31 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:26 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:26 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:42 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:42 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye

=== 2023-06-05 ===
* 09:41 arturo: [codfw1dev] rebooting bastion-codfw1dev-02 (no IP address in the main interface) [[phab:T336963|T336963]]

=== 2023-06-02 ===
* 14:40 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by arturo@nostromo
* 12:59 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 12:58 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:58 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:44 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 12:44 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:43 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-26 ===
* 16:03 andrewbogott: &quot;maintain-views --all-databases --replace-all&quot; on clouddb1021 for [[phab:T337446|T337446]]

=== 2023-05-22 ===
* 19:54 andrewbogott: deleting project 'citelearn' as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge#SHUTDOWN_citelearn

=== 2023-05-21 ===
* 23:29 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 23:27 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:44 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:44 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-19 ===
* 14:53 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-18 ===
* 21:39 andrewbogott: deleting obsolete roles '4d8cad783d6342efa8414d7d36fbc034 {{!}} projectadmin_renamed_for_[[phab:T330759|T330759]]' and 'f473273fac7146b3bdbf22e5d4504f95 {{!}} user_renamed_for_[[phab:T330759|T330759]]' on eqiad1. State pre-deletion is dumped to /root/allassignmentspredeletion.txt  on cloudcontrol1007.
* 21:35 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:33 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:18 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:10 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Restarting openstack services on cloudcontrol2001-dev@local1: ['cinder-volume'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:20 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-15 ===
* 14:28 wm-bot2: Drained cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:28 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: 96ce2ed0-3aff-4d04-be0b-{{Gerrit|e16513070617}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:27 wm-bot2: Draining cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:23 wm-bot2: Drained cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 110176f8-04d5-4110-bb7d-{{Gerrit|1ab272bd8be2}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: c6e92e13-49f4-4db3-8a13-{{Gerrit|8692ccfd3bc9}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:12 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:12 wm-bot2: Drained cloudvirt1035.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:11 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: fa730dec-848f-45fb-9eda-{{Gerrit|e74bd874c5c9}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:10 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1006: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1007: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1005: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:01 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: eb1cfac0-d481-4baa-b9cd-{{Gerrit|15e5fbcef495}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:00 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:58 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: 3e6c3ff3-7d55-4777-9032-{{Gerrit|b867a257eced}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:57 wm-bot2: Draining cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:53 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: 0693664a-df78-417e-ba34-{{Gerrit|590e5a0a9981}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:52 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:49 wm-bot2: Set cloudvirt cloudvirt1035.eqiad.wmnet maintenance (downtime id: e6929ab8-4bc3-4186-817b-{{Gerrit|9b53dbd597c6}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:48 wm-bot2: Draining cloudvirt1035.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 andrewbogott: restarting nova services in eqiad1, trying to free up db connections
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:33 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: fef43d73-6fd4-4dde-a0ac-{{Gerrit|95fd69a9b0c1}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:32 wm-bot2: Draining cloudvirt1034.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 5f867662-e824-498c-a715-{{Gerrit|2e2ad50f0bb5}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: 0f515150-1313-41d4-a5f6-{{Gerrit|9bc00ce9b245}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Draining cloudvirt1033.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Drained cloudvirt1032.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot2: Set cloudvirt cloudvirt1032.eqiad.wmnet maintenance (downtime id: e826adc3-addd-44d8-b39e-{{Gerrit|ae7bd2df1e60}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:49 wm-bot2: Draining cloudvirt1032.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:49 wm-bot2: Drained cloudvirt1031.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:14 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 4154c818-744c-4d84-9883-{{Gerrit|cae7a5826ed5}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:13 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:13 wm-bot2: Drained cloudvirt1026.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:04 wm-bot2: Set cloudvirt cloudvirt1026.eqiad.wmnet maintenance (downtime id: cdfc3d01-ec1e-483c-9a38-{{Gerrit|834193e487ff}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Draining cloudvirt1026.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Drained cloudvirt1025.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 11:51 wm-bot2: Set cloudvirt cloudvirt1025.eqiad.wmnet maintenance (downtime id: 6a56757d-35de-499e-8209-{{Gerrit|728bcf62a22a}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 11:50 wm-bot2: Draining cloudvirt1025.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-12 ===
* 17:52 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-10 ===
* 16:09 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 andrewbogott: running &quot;cookbook -c ~/.config/spicerack/cookbook_config.yaml wmcs.openstack.restart_openstack --cluster-name eqiad1 --all&quot; to pick up changes for testing [[phab:T336379|T336379]]
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 03:05 andrewbogott: &quot;systemctl restart puppet-enc&quot; on enc-1.cloudinfra.eqiad1.wikimedia.cloud.  Seems to have crashed.

=== 2023-05-05 ===
* 16:07 wm-bot2: Drained cloudvirt1024.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:03 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 95995009-09d6-496e-8cd2-{{Gerrit|0cfac93d3cf7}}, use this to unset) ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:02 wm-bot2: Draining cloudvirt1024.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Drained cloudvirt1023.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:51 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 53c46cae-00af-4664-97ff-{{Gerrit|266b393335bb}}, use this to unset) ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:50 wm-bot2: Draining cloudvirt1023.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:49 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 528ea4f6-8088-475e-937f-{{Gerrit|098ffba861b6}}, use this to unset) - cookbook ran by andrew@bullseye
* 15:47 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 3ef85b5e-d9d9-4b24-901b-{{Gerrit|a3058a7d0615}}, use this to unset) - cookbook ran by andrew@bullseye
* 15:44 andrewbogott: moved cloudvirt1023 and cloudvirt1024 from 'ceph' aggregate to 'maintenance' aggregate, prep for decom [[phab:T336064|T336064]]
* 15:44 andrewbogott: moved cloudvirt1028 from 'localdisk' aggregate to 'maintenance' aggregate. Nothing new should be scheduled here, local storage should now move to cloudvirtlocal100x
* 15:41 andrewbogott: moved cloudvirt1055 and cloudvirt1056 from 'spare' to 'ceph' aggregate.  Prep for removing two obsolete cloudvirts, 1023 and 1024.  [[phab:T336064|T336064]]

=== 2023-05-04 ===
* 22:49 andrewbogott: removed fullstack-* puppet reports on puppetmaster-02.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud and cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs to free up disk space

=== 2023-05-02 ===
* 13:01 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:31 wm-bot2: Destroying OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 12:30 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Added 1 new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Added OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus

=== 2023-05-01 ===
* 17:09 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 17:09 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 17:08 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 17:08 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 15:22 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 13:53 taavi: running wmcs-novastats-puppetleaks in real mode [[phab:T334127|T334127]]
* 13:51 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus

=== 2023-04-18 ===
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1007: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1006: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1005: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudvirt1056: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1019: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1060: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1023: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1038: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1036: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1039: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1050: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1061: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1028: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1020: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1052: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1040: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1043: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1026: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1030: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1048: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1025: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1035: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1055: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1042: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1059: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1057: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1032: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1029: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1044: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1047: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1034: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1058: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1045: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1041: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1031: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1046: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1027: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1054: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1033: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1037: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1051: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1053: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1049: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1019: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1020: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:41 andrewbogott: resetting rabbitmq on cloudrabbit1003 due to splitbrain
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1019: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1020: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye

=== 2023-04-17 ===
* 08:49 wm-bot2: Increased quotas by 9 cores, 1 instances, 16 ram ([[phab:T334695|T334695]]) - cookbook ran by dcaro@vulcanus

=== 2023-04-06 ===
* 17:03 andrewbogott: running wmcs-wikireplica-dns on cloudcontrol1005 to update tools-db dns entries

=== 2023-04-04 ===
* 17:23 andrewbogott: resetting all three rabbitmq nodes and restarting all openstack services as per https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Rabbitmq#Resetting_the_HA_setup

=== 2023-03-28 ===
* 13:56 andrewbogott: depooling cloudweb1003 before switch upgrade
* 10:58 dhinus: disabled tool &quot;wb&quot; by clicking the disable button at https://toolsadmin.wikimedia.org/tools/id/wb [[phab:T328693|T328693]]
* 08:34 arturo: cleanup neutron agents for cloudvirt1021/1022 (decom)
* 08:32 arturo: cleanup neutron agents for cloudvirt1017 (decom)

=== 2023-03-27 ===
* 14:30 wm-bot2: Drained cloudvirt1024.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:19 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 3f43d3ca-696c-4d3c-8d5b-{{Gerrit|e57984f0eb86}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:18 wm-bot2: Draining cloudvirt1024.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:17 wm-bot2: Drained cloudvirt1023.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:08 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: f4767781-ea26-453b-9521-{{Gerrit|847a8340a249}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:07 wm-bot2: Draining cloudvirt1023.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:04 wm-bot2: Drained cloudvirt1022.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:55 wm-bot2: Set cloudvirt cloudvirt1022.eqiad.wmnet maintenance (downtime id: 0e794cfe-5896-46c1-842d-{{Gerrit|c34719140d4f}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:54 wm-bot2: Draining cloudvirt1022.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:54 wm-bot2: Drained cloudvirt1021.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:48 wm-bot2: Set cloudvirt cloudvirt1021.eqiad.wmnet maintenance (downtime id: e7a904ca-003d-450a-ad85-{{Gerrit|886fb80dfc41}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:47 wm-bot2: Draining cloudvirt1021.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:46 wm-bot2: Drained cloudvirt1017.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Set cloudvirt cloudvirt1017.eqiad.wmnet maintenance (downtime id: 0ff0090f-26c3-4278-a9c9-{{Gerrit|cce518559408}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:35 wm-bot2: Draining cloudvirt1017.eqiad.wmnet - cookbook ran by andrew@bullseye

=== 2023-03-22 ===
* 12:41 taavi: delete wmde-templates-alpha project [[phab:T332773|T332773]]

=== 2023-03-08 ===
* 21:45 bd808: maintain-kubeusers container in CrashLoopBackoff, investigating
* 13:49 dcaro: stopping puppet on labostre1004 to debug maintain-dbusers

=== 2023-03-07 ===
* 16:06 andrewbogott: updated application credential roles, replacing 'user' with 'reader' and 'projectadmin' with 'member':  update application_credential_role set role_id='f75a3c410bca4e96a1cf6ac103b0ccaf' where role_id='f473273fac7146b3bdbf22e5d4504f95'  and update application_credential_role set role_id='38676f30eaeb44518bf7e144a73c8da6' where role_id='4d8cad783d6342efa8414d7d36fbc034'
* 10:11 dcaro: there was a little unavailability for some VMs while ceph was starting to rebalance things, but it seems stable and moving data around ([[phab:T331141|T331141]])
* 09:37 dcaro: Changing ceph crush map to allow rack HA on eqiad1 cluster ([[phab:T331141|T331141]])

=== 2023-03-03 ===
* 12:12 arturo: installing haproxy updates ([[phab:T331119|T331119]])

=== 2023-03-02 ===
* 16:17 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1010.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Added 1 new OSDs ['cloudcephosd1010.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Added OSD cloudcephosd1010.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot2: Finished rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:10 wm-bot2: Rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Adding OSD cloudcephosd1010.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Adding new OSDs ['cloudcephosd1010.eqiad.wmnet'] to the cluster ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 10:43 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1005.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:44 wm-bot2: Added 1 new OSDs ['cloudcephosd1005.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:44 wm-bot2: Added OSD cloudcephosd1005.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:36 wm-bot2: Finished rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:33 wm-bot2: Rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Adding OSD cloudcephosd1005.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Adding new OSDs ['cloudcephosd1005.eqiad.wmnet'] to the cluster ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-28 ===
* 22:47 andrewbogott: adding new 'member' role assignment to every user/project pair that currently has the 'user' assignment.  [[phab:T330759|T330759]]
* 09:47 wm-bot2: Depooled and destroyed OSD daemons [79, 78, 77, 76, 75, 74, 73, 72] and removed the OSD host cloudcephosd1010 from the CRUSH map. ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:46 wm-bot2: Destroying OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:28 wm-bot2: Depooling OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot2: Depooling OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot2: Depooled and destroyed OSD daemons [39, 38, 37, 36, 35, 34, 33, 32] and removed the OSD host cloudcephosd1005 from the CRUSH map. ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:11 wm-bot2: Destroying OSDs with ids in [39, 38, 37, 36, 35, 34, 33, 32] on cloudcephosd1005 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:55 wm-bot2: Depooling OSDs with ids in [39, 38, 37, 36, 35, 34, 33, 32] on cloudcephosd1005 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-27 ===
* 21:01 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1004.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:53 wm-bot2: Added 1 new OSDs ['cloudcephosd1004.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:53 wm-bot2: Added OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:45 wm-bot2: Finished rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:42 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:41 wm-bot2: Adding OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:41 wm-bot2: Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Adding OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:14 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1003.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:13 wm-bot2: Added 1 new OSDs ['cloudcephosd1003.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:13 wm-bot2: Added OSD cloudcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:05 wm-bot2: Finished rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:02 wm-bot2: Rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding OSD cloudcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding new OSDs ['cloudcephosd1003.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:59 wm-bot2: Adding OSD coludcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:59 wm-bot2: Adding new OSDs ['coludcephosd1003.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:58 wm-bot2: Adding OSD coludcephosd1003... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:58 wm-bot2: Adding new OSDs ['coludcephosd1003'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-22 ===
* 08:25 wm-bot2: Depooled and destroyed OSD daemons [31, 30, 29, 28, 27, 26, 25, 24] and removed the OSD host cloudcephosd1004 from the CRUSH map. ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:22 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:15 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:13 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 07:23 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 07:02 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-21 ===
* 20:27 andrewbogott: deleted 200 more orphaned VM images with wmcs-novastats-cephleaks
* 17:18 andrewbogott: shutting down postgres on clouddb1004/1003, then shutting down the vms
* 15:50 wm-bot2: Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:48 wm-bot2: Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 14:00 wm-bot2: Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-16 ===
* 19:14 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1002.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:55 dcaro: Manually zapped /dev/sdc on cloudcephosd1002, probably a leftover drive since the beginning (or during the reimage the drives changed names, and this one had leftovers from the previous OS) ([[phab:T329498|T329498]])
* 17:47 wm-bot2: Added 1 new OSDs ['cloudcephosd1002.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:47 wm-bot2: Added OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:42 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:42 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:50 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:50 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:00 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:00 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot2: Added 1 new OSDs ['cloudcephosd1001.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Destroying OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot2: Depooling OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:20 wm-bot2: Destroying OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:19 wm-bot2: Depooling OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:03 wm-bot2: Destroying OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:01 wm-bot2: Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 10:59 wm-bot2: Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 10:15 dcaro: purges osd daemons 48 and 40 from eqiad ceph cluster ([[phab:T329709|T329709]])

=== 2023-02-15 ===
* 14:53 andrewbogott: deleting another 100 leaked VM images with wmcs-novastats-cephleaks
* 13:39 wm-bot2: Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:13 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:11 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:11 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot2: Destroying OSDs with id [[12345]] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:09 wm-bot2: Destroying OSDs with id ['12345'] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus

=== 2023-02-14 ===
* 13:17 andrewbogott: restarting all eqiad1 openstack services because that seems to sometimes help things *shrug*

=== 2023-02-13 ===
* 14:06 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 8fbf6bfd-eec1-4d81-8e0d-{{Gerrit|ea431d8411ee}} ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:32 taavi: re-enable puppet on labstore1004 [[phab:T329377|T329377]]

=== 2023-02-09 ===
* 21:17 andrewbogott: deleted 10% of leaked VM ceph images using wmcs-novastats-cephleaks (only 10% out of an abundance of caution)

=== 2023-02-08 ===
* 17:08 arturo: changing to cloudgw network setup, make VIPs /32 ([[phab:T295774|T295774]])

=== 2023-02-07 ===
* 11:26 arturo: [codfw1dev] testing network changes in cloudgw, expect unrealiable network ([[phab:T295774|T295774]])

=== 2023-02-04 ===
* 13:44 taavi: drop old columns from oathauth_users table on labtestwiki [[phab:T328131|T328131]]

=== 2023-02-03 ===
* 15:00 andrewbogott: restarted nova services in eqiad1 in an attempt to eke out another day or two of stability
* 14:13 taavi: attached GrapheSuppression developer account to wikitech

=== 2023-02-02 ===
* 13:14 dcaro_away: draining osd.48 from node cloudcephosd1001 ([[phab:T316544|T316544]])
* 12:57 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 7ac2b25a-d1bb-4789-8aa6-{{Gerrit|b9435b505349}} ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus

=== 2023-01-30 ===
* 22:34 wm-bot2: Upgraded and rebooted host cloudrabbit1002.wikimedia.org - cookbook ran by andrew@bullseye
* 21:34 andrewbogott: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/884922 and upgrading rabbitmq nodes for [[phab:T328155|T328155]]

=== 2023-01-27 ===
* 20:08 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:22 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 15:25 andrewbogott: restarting openstack services in eqiad1, another attempt to address instability

=== 2023-01-26 ===
* 20:34 andrewbogott: shutting down mariadb on cloudbackup2001-dev, testing the waters for [[phab:T328079|T328079]]

=== 2023-01-22 ===
* 03:42 andrewbogott: reset eqiad1 rabbitmq in an attempt to resolve some mild instability

=== 2023-01-20 ===
* 15:26 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 15:26 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['f47a3d91-b270-4c90-acc8-d85075a6bf8e'], use this to unset) - cookbook ran by andrew@bullseye
* 13:15 arturo: reinstall python3-neutron (to reset manual patching) on all cloudnet nodes and patch it via puppet, then restart neutron-l3-agent by hand ([[phab:T327463|T327463]])
* 10:12 arturo: [codfw1dev] failover neutron-l3-agent between cloudnet2005-dev/cloudnet2006-dev a couple of times [[phab:T327463|T327463]]
* 02:17 andrewbogott: stopping neutron-l3-agent on cloudnet1005 because it's logging at a furious rate and about to fill the drive

=== 2023-01-19 ===
* 18:06 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 18:06 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['36d5af6a-7d8e-4d0c-831e-1bf05c255984'], use this to unset) - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 17:22 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['66ec4f04-d25b-4067-be9e-2fe12cb1d3ff'], use this to unset) - cookbook ran by andrew@bullseye

=== 2023-01-18 ===
* 22:32 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 347cb75e-215e-4b85-ae14-{{Gerrit|4ce1934c70c7}}, use this to unset) - cookbook ran by andrew@bullseye
* 22:01 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 9bf6212f-7fdb-4869-8190-{{Gerrit|b07387b2bc7e}}, use this to unset) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 11aec8ea-f443-41ae-b79a-{{Gerrit|5e1e3aa94546}}, use this to unset) - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:03 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:38 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:38 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:37 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:37 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-01-17 ===
* 19:32 wm-bot2: Upgraded and rebooted host cloudbackup2002.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:04 wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye
* 17:52 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 17:23 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye

=== 2023-01-13 ===
* 17:36 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:26 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:26 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:25 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:21 wm-bot2: Restarting openstack services: &lt;nowiki&gt;{&lt;/nowiki&gt;'cloudcontrol2001-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'], 'cloudcontrol2004-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'], 'cloudvirt2001-dev': ['nova-compute'], 'cloudvirt2002-dev': ['nova-compute'], 'cloudvirt2003-dev': ['nova-compute'], 'cloudcontrol2005-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata
* 10:41 arturo: restart backup_vm.service on cloudbackup1003/1004 to recover from a nova Unknown Error (HTTP 503)

=== 2023-01-12 ===
* 22:34 andrewbogott: updated the Bullseye base image with the upstream {{Gerrit|20221219}} build
* 14:12 wm-bot2: Created project checkuser-beta-wiki with default quotas. ([[phab:T326740|T326740]]) - cookbook ran by arturo@nostromo

=== 2023-01-06 ===
* 18:42 wm-bot2: Safe reboot of cloudvirt2003-dev.codfw.wmnet finished successfully - cookbook ran by andrew@bullseye
* 18:42 wm-bot2: unset cloudvirt2003-dev.codfw.wmnet maintenance (aggregates: ceph) - cookbook ran by andrew@bullseye
* 18:39 wm-bot2: Drained cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: b50d9d3a-4f1d-4522-b86f-{{Gerrit|722fc9c55c87}}, use this to unset) - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Safe rebooting cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Draining cloudvirt2003-dev.cdofw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Safe rebooting cloudvirt2003-dev.cdofw.wmnet - cookbook ran by andrew@bullseye
* 00:14 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 00:08 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 00:01 wm-bot2: Upgraded and rebooted host cloudnet2006-dev.codfw.wmnet - cookbook ran by andrew@bullseye

=== 2023-01-05 ===
* 23:54 wm-bot2: Upgraded and rebooted host cloudnet2005-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 23:38 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 23:25 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 23:13 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 22:18 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 22:10 andrewbogott: upgrading codfw1dev openstack to version 'zed'

=== 2023-01-04 ===
* 21:18 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org - cookbook ran by andrew@bullseye
* 21:11 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org - cookbook ran by andrew@bullseye
* 20:12 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org - cookbook ran by andrew@bullseye
* 20:04 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org - cookbook ran by andrew@bullseye
* 14:45 wm-bot2: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] - cookbook ran by fran@wmf3169
* 14:44 wm-bot2: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:41 wm-bot2: Rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:41 wm-bot2: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:38 wm-bot2: Rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:37 wm-bot2: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev - cookbook ran by fran@wmf3169

=== 2023-01-03 ===
* 22:11 wm-bot2: Upgraded and rebooted host cloudservices2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Upgraded and rebooted host cloudservices2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 15:25 taavi: restart designate-sink everywhere to pick up wmf-sink changes

=== 2022-12-25 ===
* 14:21 taavi: register developer account 'instance-puppet-user-dev' to update the codfw1dev instance-puppet repo without access to the eqiad1 repo [[phab:T318504|T318504]]

=== 2022-12-22 ===
* 15:16 dcaro: added submit rights for JenkinsBot on all cloud/* gerrit repos

=== 2022-12-21 ===
* 04:59 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:59 wm-bot2: Finished rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:35 wm-bot2: Rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:35 wm-bot2: Finished rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:12 wm-bot2: Rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:12 wm-bot2: Finished rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:48 wm-bot2: Rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:48 wm-bot2: Finished rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:24 wm-bot2: Rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:24 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:00 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:00 wm-bot2: Finished rebooting node cloudcephosd1024.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:36 wm-bot2: Rebooting node cloudcephosd1024.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:36 wm-bot2: Finished rebooting node cloudcephosd1023.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:12 wm-bot2: Rebooting node cloudcephosd1023.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:12 wm-bot2: Finished rebooting node cloudcephosd1022.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:47 wm-bot2: Rebooting node cloudcephosd1022.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:47 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:22 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:22 wm-bot2: Finished rebooting node cloudcephosd1020.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:58 wm-bot2: Rebooting node cloudcephosd1020.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:58 wm-bot2: Finished rebooting node cloudcephosd1019.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:34 wm-bot2: Rebooting node cloudcephosd1019.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:34 wm-bot2: Finished rebooting node cloudcephosd1018.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:09 wm-bot2: Rebooting node cloudcephosd1018.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:09 wm-bot2: Finished rebooting node cloudcephosd1017.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye

=== 2022-12-20 ===
* 23:45 wm-bot2: Rebooting node cloudcephosd1017.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:45 wm-bot2: Finished rebooting node cloudcephosd1016.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:21 wm-bot2: Rebooting node cloudcephosd1016.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:21 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:56 wm-bot2: Rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:55 wm-bot2: Finished rebooting node cloudcephosd1014.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Rebooting node cloudcephosd1014.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Finished rebooting node cloudcephosd1013.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:08 andrewbogott: restarting openstack services with wmcs.openstack.restart_openstack due to miscellaneous trove failures
* 22:06 wm-bot2: Rebooting node cloudcephosd1013.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:06 wm-bot2: Finished rebooting node cloudcephosd1012.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Rebooting node cloudcephosd1012.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Finished rebooting node cloudcephosd1011.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Rebooting node cloudcephosd1011.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Finished rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Finished rebooting node cloudcephosd1009.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:26 wm-bot2: Rebooting node cloudcephosd1009.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:26 wm-bot2: Finished rebooting node cloudcephosd1008.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:01 wm-bot2: Rebooting node cloudcephosd1008.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:01 wm-bot2: Finished rebooting node cloudcephosd1007.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Rebooting node cloudcephosd1007.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Finished rebooting node cloudcephosd1006.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Rebooting node cloudcephosd1006.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Finished rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:45 wm-bot2: Rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:45 wm-bot2: Finished rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:20 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:20 wm-bot2: Finished rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:56 wm-bot2: Rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:56 wm-bot2: Finished rebooting node cloudcephosd1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Rebooting node cloudcephosd1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Finished rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:07 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:07 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 16:49 wm-bot2: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:48 wm-bot2: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:46 wm-bot2: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:46 wm-bot2: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 16:39 wm-bot2: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:39 wm-bot2: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:41 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:41 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 15:40 wm-bot2: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:40 wm-bot2: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:37 wm-bot2: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:37 wm-bot2: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:30 wm-bot2: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:30 wm-bot2: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:29 wm-bot2: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:29 wm-bot2: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:25 wm-bot2: Rebooting node cloudcephmon2006-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:25 wm-bot2: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Rebooting node cloudcephmon2005-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:19 wm-bot2: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:19 wm-bot2: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:18 wm-bot2: Finished rebooting the nodes ['cloudcephmon1001', 'cloudcephmon1002', 'cloudcephmon1003'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:18 wm-bot2: Finished rebooting node cloudcephmon1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:15 wm-bot2: Rebooting node cloudcephmon1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:15 wm-bot2: Finished rebooting node cloudcephmon1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Finished rebooting node cloudcephmon1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:08 wm-bot2: Rebooting node cloudcephmon1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:08 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye

=== 2022-12-17 ===
* 07:50 taavi: deleted project packagist-mirror per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge#packagist-mirror

=== 2022-12-16 ===
* 19:36 volans: restarted sshd twice on bastion-restricted-eqiad1-02 to debug SSH connections for [[phab:T319401|T319401]]
* 08:46 dcaro: restart designate-sink on both cloudservice hosts ([[phab:T322279|T322279]])
* 08:45 dcaro: restart designate-sink on both cloudservice hosts

=== 2022-12-07 ===
* 22:07 andrewbogott: systemctl restart libvirt-guests.service  on cloudvirt1019 to get ceph/rbd working on VMS on this hypervisor

=== 2022-12-03 ===
* 19:24 taavi: restart designate-sink on both cloudservices hosts

=== 2022-11-30 ===
* 20:03 andrewbogott: changing all rabbitmq queues to quorum queues.  Will be noisy!  [[phab:T318816|T318816]]
* 02:54 wm-bot2: Upgraded and rebooted host cloudbackup2002.codfw.wmnet - cookbook ran by andrew@bullseye

=== 2022-11-28 ===
* 13:00 wm-bot2: unset cloudvirt1043.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 10:28 wm-bot2: Drained cloudvirt1043.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:19 wm-bot2: Set cloudvirt cloudvirt1043.eqiad.wmnet maintenance (downtime id: bb94dd24-fef9-4c9c-8f79-{{Gerrit|b6e15023ce69}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:18 wm-bot2: Draining cloudvirt1043.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-25 ===
* 10:54 wm-bot2: deleted VM canary2001-dev-2 from cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 10:54 wm-bot2: created VM canary2001-dev-3 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 10:53 wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:5b2ca632-2ea0-4007-9b40-{{Gerrit|4f84f8e2428b}}) - cookbook ran by arturo@nostromo
* 10:46 wm-bot2: Created new flavor: g3.cores1.ram1.disk20.admin (id:fecbd56d-0969-45f3-80fd-{{Gerrit|2b463a5b6270}}) - cookbook ran by arturo@nostromo

=== 2022-11-24 ===
* 16:53 wm-bot2: deleted VM canary2001-dev-1 from cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:53 wm-bot2: created VM canary2001-dev-2 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2003-dev-1 in cloudvirt2003-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2002-dev-1 in cloudvirt2002-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2001-dev-1 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:36 wm-bot2: Created new flavor: cloudvirt-canary-ceph (id:0d06701a-2845-4298-b2b4-{{Gerrit|fabf8b1ddcbb}}) - cookbook ran by arturo@nostromo
* 13:03 wm-bot2: unset cloudvirt1044.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 11:51 wm-bot2: Drained cloudvirt1044.eqiad.wmnet - cookbook ran by arturo@nostromo
* 11:37 wm-bot2: Set cloudvirt cloudvirt1044.eqiad.wmnet maintenance (downtime id: 10076ecc-0f94-4d56-9bbd-{{Gerrit|bebb48bdc126}}, use this to unset) - cookbook ran by arturo@nostromo
* 11:35 wm-bot2: Draining cloudvirt1044.eqiad.wmnet - cookbook ran by arturo@nostromo
* 10:16 dcaro: removed ip6 dns name entry from nb for coluddb* ([[phab:T323550|T323550]])
* 09:53 dcaro: removed ip6 dns entry from nb for coluddb1013 ([[phab:T323550|T323550]])

=== 2022-11-23 ===
* 15:00 wm-bot2: unset cloudvirt1045.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Drained cloudvirt1045.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:35 wm-bot2: Set cloudvirt cloudvirt1045.eqiad.wmnet maintenance (downtime id: 2386b468-0f21-4ecb-91e2-{{Gerrit|e19ace66881d}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:34 wm-bot2: Draining cloudvirt1045.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:20 wm-bot2: unset cloudvirt1046.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 12:17 wm-bot2: Drained cloudvirt1046.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:04 wm-bot2: Set cloudvirt cloudvirt1046.eqiad.wmnet maintenance (downtime id: 6291d38e-c04c-4aa0-88da-{{Gerrit|2b329874a9b9}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:03 wm-bot2: Draining cloudvirt1046.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:02 wm-bot2: unset cloudvirt1047.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 11:32 arturo: [codfw1dev] created project cloudvirt-canary
* 10:13 wm-bot2: Drained cloudvirt1047.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:01 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 2c7ccc17-2be3-427d-aaec-{{Gerrit|57fadca0de5b}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:00 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-22 ===
* 13:26 wm-bot2: unset cloudvirt1048.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 12:15 wm-bot2: unset cloudvirt1049.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 10:58 wm-bot2: Drained cloudvirt1049.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:37 wm-bot2: Set cloudvirt cloudvirt1049.eqiad.wmnet maintenance (downtime id: 3ae0a20d-3cf0-4eba-bea6-{{Gerrit|45aa61d8ad00}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:36 wm-bot2: Draining cloudvirt1049.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:21 wm-bot2: Unset cloudvirt cloudvirt1050.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo

=== 2022-11-21 ===
* 16:19 wm-bot2: Drained cloudvirt1050.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:11 wm-bot2: Set cloudvirt cloudvirt1050.eqiad.wmnet maintenance (downtime id: 0b154a93-a9d3-4ac7-bcfc-{{Gerrit|b67c49abe97b}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:09 wm-bot2: Draining cloudvirt1050.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:07 wm-bot2: Unset cloudvirt cloudvirt1051.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 15:18 wm-bot2: Drained cloudvirt1051.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 15:02 wm-bot2: Set cloudvirt cloudvirt1051.eqiad.wmnet maintenance (downtime id: 1de1174b-ec46-47a3-911c-{{Gerrit|b5808ce37028}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 15:00 wm-bot2: Draining cloudvirt1051.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 14:57 wm-bot2: Unset cloudvirt cloudvirt1052.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 12:53 wm-bot2: Drained cloudvirt1052.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:30 wm-bot2: Set cloudvirt cloudvirt1052.eqiad.wmnet maintenance (downtime id: 30db8a9b-08db-456f-8106-{{Gerrit|53188ff5f989}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:29 wm-bot2: Draining cloudvirt1052.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:15 wm-bot2: Unset cloudvirt cloudvirt1053.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 10:13 arturo: drained cloudvirt1053 in preparation for reimage (was spare anyway)

=== 2022-11-18 ===
* 13:37 arturo: [codfw1dev] reimaged cloudvirt2001-dev and cloudvirt2002-dev
* 11:36 wm-bot2: Set cloudvirt cloudvirt2001-dev.codfw.wmnet maintenance (downtime id: 2fb9df34-fc2d-45b9-b21f-{{Gerrit|c0ec09008b92}}, use this to unset) - cookbook ran by arturo@nostromo
* 11:36 wm-bot2: Draining cloudvirt2001-dev.codfw.wmnet - cookbook ran by arturo@nostromo

=== 2022-11-16 ===
* 20:07 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:56 wm-bot2: Upgraded and rebooted host cloudservices2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:16 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 18:59 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 12:52 arturo: failovered cloudgw1002 into cloudgw1001 for reimage, IRC bots were briefly disconnected

=== 2022-11-14 ===
* 20:22 wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye
* 20:07 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 19:42 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye
* 19:28 andrewbogott: beginning OpenStack upgrade in eqiad1 -- [[phab:T305828|T305828]]
* 19:22 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 19:14 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 19:08 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 12:52 arturo: cleanup old network vlan interface names from /etc/network/interfaces in cloudnet1005/1006

=== 2022-11-11 ===
* 13:24 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: edad3915-b7c6-4b23-bb9c-{{Gerrit|ab13b04a41c5}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:23 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:17 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: a09a9868-8aae-4dc0-8510-{{Gerrit|a3923b703060}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:16 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-10 ===
* 16:19 wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: 346013ec-ce4e-497e-ad65-{{Gerrit|2d215b14998c}}, use this to unset). ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:18 wm-bot2: Draining 'cloudvirt2002-dev.codfw.wmnet'. ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:13 wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: a40a8487-22ee-4bc6-bbe4-{{Gerrit|694a615e3bf5}}, use this to unset). ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-08 ===
* 11:17 taavi: backfilling security groups for metricsinfra access on all projects [[phab:T288108|T288108]]

=== 2022-11-07 ===
* 21:01 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 20:40 andrewbogott: upgrading eqiad1 designate to version 'yoga'

=== 2022-11-04 ===
* 17:57 andrewbogott: removing cinderv2 API endpoints from keystone catalog; this is deprecated and removed in Yoga.  prep for [[phab:T305828|T305828]]

=== 2022-11-03 ===
* 19:24 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 00:08 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye

=== 2022-11-02 ===
* 23:39 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:38 wm-bot2: Upgraded and rebooted host cloudnet2006-dev.codfw.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Upgraded and rebooted host cloudnet2005-dev.codfw.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:13 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:00 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 22:54 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 20:29 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye

=== 2022-10-31 ===
* 13:09 arturo: restart keepalived on all 4 cloudgw servers to run them with `-D` in /etc/default/keepalived to further debug [[phab:T320975|T320975]]

=== 2022-10-26 ===
* 16:18 wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:bf48880d-0c1b-4c2a-8e8b-{{Gerrit|778d28b16561}}) ([[phab:T319446|T319446]]) - cookbook ran by dcaro@vulcanus
* 09:34 taavi: running wmcs-puppetcertleaks in delete mode
* 09:09 taavi: running wmcs-novastats-dnsleaks in delete mode

=== 2022-10-25 ===
* 16:03 arturo: [codfw1dev] [[phab:T321220|T321220]] root@cloudcontrol2001-dev:~# openstack subnet create magnum --no-dhcp --network 57017d7c-3817-429a-8aa3-{{Gerrit|b028de82cdcc}} --ip-version 4 --gateway auto --subnet-range 192.168.0.0/24
* 14:38 arturo: [codfw1dev] restart neutron-l3-agent in cloudnet2005-dev, it was dead after rabbit connectivity problems

=== 2022-10-24 ===
* 18:50 wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Finished rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:47 wm-bot2: Rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:47 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by andrew@bullseye
* 18:38 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by andrew@bullseye
* 18:21 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:21 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl

=== 2022-10-20 ===
* 23:23 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:23 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:20 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:20 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:19 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:16 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:07 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:07 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:06 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:06 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 101c41d1-d65d-4088-b6d2-{{Gerrit|eac859e45ef8}}, use this to unset). - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:01 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:57 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 7197ce34-cc57-4677-a1a9-{{Gerrit|05e20cd0dd80}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:38 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 68e4cc1d-9def-444e-84a7-{{Gerrit|21b0b5adfa72}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 73388c1e-369b-40af-a2c1-{{Gerrit|96936128c324}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 12feeca2-ea59-48e9-9235-{{Gerrit|1cecf5b384cd}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 1b34587f-2770-4d2e-bb31-{{Gerrit|c8ad14632d39}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:24 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:13 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: e75b00eb-7f58-4821-8139-{{Gerrit|3dfc6e97a92a}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 9151511e-bcc0-4367-a976-{{Gerrit|7cff060308aa}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 241c0bd9-c3cf-40e4-95dc-{{Gerrit|9b51d9823fe9}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 7a53c274-1d5f-4c94-9a0d-{{Gerrit|721c3e8f7239}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:10 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:10 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:06 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:54 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:54 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:51 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 0b75b44c-1efe-4edf-8f5e-{{Gerrit|42a67e8d3b13}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: aeb9c3a6-961a-4873-85d4-{{Gerrit|929248aebb8b}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: eb14cc04-3293-4cf8-a46f-{{Gerrit|1f87d8b0bcc4}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 84a3cd99-2643-495a-86b6-{{Gerrit|7ae80b11a30d}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:32 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:32 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:28 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:28 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 01a0b69c-2e27-4331-9635-{{Gerrit|403a668dac29}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:27 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:27 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:23 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: ea29f65f-6d86-4568-8db9-{{Gerrit|a85faa827447}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:22 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:22 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:18 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:18 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:16 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:14 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: d6b6bb4d-c7c0-4bd2-9a02-{{Gerrit|dee9a5ec6a3e}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 4e109149-7d67-403f-8b21-{{Gerrit|f829235ea491}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: 341f4b8c-6d6e-4190-9f2d-{{Gerrit|a1a1483abadd}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: f76e8f14-920e-4d51-a44e-{{Gerrit|321a55dcb0c5}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:56 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:56 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:54 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:35 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:34 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 0b0d8090-4d31-4d43-8545-{{Gerrit|c25209e2ef58}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:31 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 3d928554-a79c-419a-bcc4-{{Gerrit|c8d63791d8e7}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 3d3c8aa5-45f7-429e-a9d7-{{Gerrit|b5181b29dc13}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: b9027020-8a90-4c40-b0e6-{{Gerrit|6c8767f87917}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: f0d60fab-64e9-4da2-826c-{{Gerrit|229d31d0fbc2}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:16 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:14 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:11 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: b65da7e2-9fd2-4a1e-8dd4-{{Gerrit|88ca65936ae4}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: cbbf114c-9a4c-4dd2-9b58-{{Gerrit|50da8bd896ca}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 5969beaf-72bf-4af9-ae4d-{{Gerrit|8e5c3331e78e}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: b8f7389d-79b4-411d-b3d5-{{Gerrit|ec8f92eba101}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:53 wm-bot2: Safe reboot of 'cloudvirt1051.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:53 wm-bot2: Unset cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:51 wm-bot2: Safe reboot of 'cloudvirt1049.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:51 wm-bot2: Unset cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:50 wm-bot2: Safe reboot of 'cloudvirt1048.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:50 wm-bot2: Unset cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Drained 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: c5dbfa7b-72fc-4156-8257-{{Gerrit|af224a725b78}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: c1a3e92c-cb04-46e4-ac5d-{{Gerrit|784c79601b05}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance (downtime id: 62fdc4ee-c8d5-4892-aeb9-{{Gerrit|a681cdcbc84b}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 9ec407cd-5c00-407a-a57d-{{Gerrit|794f6c68f947}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Set cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance (downtime id: 712683ea-d58f-484b-8efb-{{Gerrit|89d1c21aa0e3}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Safe rebooting 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Draining 'cloudvirt1049.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Safe rebooting 'cloudvirt1049.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Draining 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Safe rebooting 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:25 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:25 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:21 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:04 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 18fae8d8-7353-4f67-90d7-{{Gerrit|8df9b3fb1ccb}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:03 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:03 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:01 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 3d3cffa3-abc6-4901-83d9-{{Gerrit|3bcc4b02fd3c}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:00 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:00 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:54 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:54 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:46 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:46 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye

=== 2022-10-15 ===
* 17:38 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki Slevinski # [[phab:T320867|T320867]]

=== 2022-10-13 ===
* 12:19 wm-bot2: OSDs (['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034'] ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 11:30 wm-bot2: OSDs (['cloudcephosd1025', 'cloudcephosd1026']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus

=== 2022-10-10 ===
* 14:01 dcaro: test2
* 01:22 andrewbogott: restarting designate-sink on cloudservices100[45], possible example of [[phab:T316614|T316614]]

=== 2022-10-09 ===
* 12:04 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki DatGuy # [[phab:T320301|T320301]]

=== 2022-10-07 ===
* 13:40 andrewbogott: dhinus is resetting rabbitmq cluster in an attempt to resolve a suspected (by Andrew) split-brain
* 11:33 arturo: rabbitmq-server.service @ cloudrabbit1002 is again up and running ([[phab:T320232|T320232]])
* 10:24 arturo: stopping rabbitmq-server.service @ cloudrabbit1002 ([[phab:T320232|T320232]])
* 10:19 arturo: restarting nova-conductor in all 3 cloudcontrols ([[phab:T320232|T320232]])
* 09:45 arturo: restarting rabbitmq-server.service @ cloudrabbit1002 ([[phab:T320232|T320232]])

=== 2022-10-06 ===
* 15:55 arturo: cloudnet1005 &amp; cloudnet1006 now in service. Secom cloudnet1003 &amp; cloudnet1004. Drop neutron agents, etc. ([[phab:T316284|T316284]])
* 11:54 arturo: rebooting cloudnet1005/1006 to see if they have the right network config ([[phab:T316284|T316284]])
* 11:50 arturo: set neutron l3 agents on cloudnet1005/1006 as down `root@cloudcontrol1005:~# neutron agent-update --admin-state-down &lt;uuid&gt;` ([[phab:T316284|T316284]])
* 11:40 arturo: [codfw1dev] rebooting both network nodes to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/839492
* 10:14 arturo: [codfw1dev] restart neutron-l3-agent on cloudnet2006-dev, it was dead

=== 2022-10-05 ===
* 14:40 wm-bot2: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T319418|T319418]]) - cookbook ran by fran@wmf3169
* 14:40 wm-bot2: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T319418|T319418]]) - cookbook ran by fran@wmf3169
* 14:28 arturo: adding cloudinstances2b-gw router to l3 agents on cloudnet1005/1006 ([[phab:T316284|T316284]])
* 13:11 wm-bot2: Added 1 new OSDs ['cloudcephosd1034.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 13:11 wm-bot2: Added OSD cloudcephosd1034.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 13:02 wm-bot2: Finished rebooting node cloudcephosd1034.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Rebooting node cloudcephosd1034.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Adding OSD cloudcephosd1034.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Adding new OSDs ['cloudcephosd1034.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169

=== 2022-10-04 ===
* 16:40 wm-bot2: Added 1 new OSDs ['cloudcephosd1033.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 16:40 wm-bot2: Added OSD cloudcephosd1033.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:20 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 14:17 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:59 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:56 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:55 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:55 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169

=== 2022-09-30 ===
* 14:52 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169
* 14:52 wm-bot2: Added OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:48 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:48 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 64eac5c6-4b1d-4269-98fd-{{Gerrit|8e5bed42ce40}}, use this to unset). ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:09 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:05 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: fb99c967-b974-4314-a2fa-{{Gerrit|31ed0e883dd3}}, use this to unset). - cookbook ran by andrew@buster
* 14:04 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 13:30 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169
* 13:30 wm-bot2: Added OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:26 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:26 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 13:16 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:16 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 13:08 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:08 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 12:45 wm-bot2: Finished rebooting node cloudcephosd1031.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:42 wm-bot2: Rebooting node cloudcephosd1031.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:41 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:41 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:26 arturo: [codfw1dev] cloudnet2005/2006-dev are now on a single NIC setup ([[phab:T318824|T318824]])
* 11:44 arturo: sysctl change (cleanup) on cloudnet1003/1004

=== 2022-09-27 ===
* 10:48 wm-bot2: Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:35 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:26 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:23 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:22 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:22 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:17 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:09 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169

=== 2022-09-26 ===
* 18:37 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 18:37 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:33 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:31 andrewbogott: rebooting cloudvirt1028 for [[phab:T317391|T317391]]
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance (downtime id: 6d1c4c53-76b9-493f-8c44-{{Gerrit|eb0413dfb9d0}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance (downtime id: 6bb80b65-616c-4e45-be4f-{{Gerrit|d2cdd8abb2bf}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance (downtime id: a3bba7e7-bcd3-482d-a486-{{Gerrit|06b6f53fd899}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:18 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8fb59651-fd42-4aee-a978-{{Gerrit|36bc7f79b4d5}}, use this to unset). - cookbook ran by andrew@buster
* 18:18 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:17 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:01 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:01 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:58 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:51 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:51 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:47 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:47 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:47 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:45 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:38 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: abfc35bb-331e-4d7e-bb92-{{Gerrit|35e675abce32}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:34 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:33 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: d4fe6ea8-76eb-45f7-b6cf-{{Gerrit|66f54ba9801c}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:33 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: c0671bda-9aae-4960-85be-{{Gerrit|58aaa9c8e021}}, use this to unset). - cookbook ran by andrew@buster
* 17:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:32 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:32 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:28 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:28 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:24 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: be6559f9-4d72-4492-b9b9-{{Gerrit|23c636d6554e}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:23 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:23 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:21 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 237dfb1b-5382-408d-aa35-{{Gerrit|1aa1cfe4c5af}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 088b97cd-0cf2-4c27-a67f-{{Gerrit|077ffa1c1c5e}}, use this to unset). - cookbook ran by andrew@buster
* 17:16 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:16 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 91625000-aa6d-4887-bf34-{{Gerrit|ef7785a4af4a}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:16 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:15 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:15 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:15 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:15 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 7364c4fe-9f6b-4539-b6fa-{{Gerrit|1e768b602a1b}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:14 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:14 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:12 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:12 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: cb8f8a21-73c0-4f54-9412-{{Gerrit|074d82582cb4}}, use this to unset). - cookbook ran by andrew@buster
* 17:11 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:11 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:09 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: d23dabd2-3bfc-4ce0-9533-{{Gerrit|cd1cf910f8e1}}, use this to unset). - cookbook ran by andrew@buster
* 17:08 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:08 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:05 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 4d7ddae1-f621-4dd5-8616-{{Gerrit|29b7699b692f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 4628dd6f-5b49-417f-b6ab-{{Gerrit|5c41992192b4}}, use this to unset). - cookbook ran by andrew@buster
* 17:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: c95529cf-b2a5-4553-bd6e-{{Gerrit|6ad4dcb2a105}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:01 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 36c68112-a964-4a7c-a093-{{Gerrit|8a6d481dea7d}}, use this to unset). - cookbook ran by andrew@buster
* 17:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:00 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:00 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:49 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:49 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:45 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:31 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 1244c159-65bb-476e-a702-{{Gerrit|8f43c253e499}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:30 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:30 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:22 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 13:19 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 13:18 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:18 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 12:32 dcaro: Changed the collation of labsdbaccount db to utf8mb4_bin ([[phab:T318047|T318047]])
* 10:14 arturo: deployed new version of maintain-dbusers ([[phab:T318047|T318047]])

=== 2022-09-25 ===
* 15:06 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:06 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: a050e47f-3a63-41ff-accb-{{Gerrit|3993c1e8b593}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:05 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:05 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:58 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:58 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 1a49d773-4dc9-4a6f-bd49-{{Gerrit|8663db77ce66}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:57 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:57 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:54 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: ab52dd42-68a6-4159-b345-{{Gerrit|e5625d209b57}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:53 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:53 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 10337415-3095-4834-8538-{{Gerrit|b3a64bd6b18d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:53 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:53 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 4cdbed3a-3775-4913-8c3b-{{Gerrit|afd57ae1ca55}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-24 ===
* 17:37 andrewbogott: restarting neutron api on cloudcontrol1006; cause of outage unknown
* 17:35 andrewbogott: restarting neutron-linuxbridge-agent on cloudvirt1022

=== 2022-09-22 ===
* 15:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:10 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04849437-a304-45a1-a037-{{Gerrit|538741a2f801}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:09 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:09 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:07 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:07 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:04 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 8810a919-61e7-4ba5-af7f-{{Gerrit|c41d1e1c8c8b}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:59 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:59 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:56 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: b6f8e0b6-f35c-41fd-8035-{{Gerrit|6efde61db3a3}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 784bff2c-8160-44dc-9bcf-{{Gerrit|ff23055a0527}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:40 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:40 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:39 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:38 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:38 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 21152e9e-bc67-4024-b68e-{{Gerrit|fd671c398642}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: f07ddb1d-e8da-43f3-8140-{{Gerrit|bc66789d37c8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:36 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:36 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:33 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:33 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:30 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 4e4b03fc-e3f1-440e-9097-{{Gerrit|cce5edaa1f08}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: d4bb2160-f492-4ddc-a5b7-{{Gerrit|eeee37007d14}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9bf00a6a-3697-4201-9a6e-{{Gerrit|76d499eccfa1}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:13 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:13 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:47 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f2888490-1804-4b23-b7b0-{{Gerrit|677853fb9ef7}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8a3e8b0a-ced3-4667-a121-{{Gerrit|66df673c7ed8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:43 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 35c761b3-f0de-4b23-9b11-{{Gerrit|2ed2e474c89f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 6121a0e1-754a-47fa-b51b-{{Gerrit|265f6386f396}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:40 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:40 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:39 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 189ae1bb-c69d-4eb0-aee9-{{Gerrit|90ab1f492aa8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:38 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:38 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:36 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: acc53776-68bf-47a4-bb82-{{Gerrit|c571b3df2945}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:31 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 07770ae7-3e7e-4615-8174-{{Gerrit|513767f95b1d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:30 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:30 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 7ae20a7d-8bff-4bc2-91c5-{{Gerrit|b0005c8164d1}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: b596f855-8019-49d2-9657-{{Gerrit|0d513ad7acb5}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:25 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 9172c785-d2d6-46e5-bed3-{{Gerrit|2f49d1033f78}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:23 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:23 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:19 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 85b4e0e2-e635-485f-9ce5-{{Gerrit|341981b3887f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 9494c283-02c8-42da-98c3-{{Gerrit|139b4d119ef8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: d181f301-249f-4b5a-bd85-{{Gerrit|47d87262d0ed}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-20 ===
* 21:02 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 21:02 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:58 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:58 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 1c4246a4-9cee-4423-8cd1-{{Gerrit|4f52f61d503f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:53 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:50 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:50 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:49 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 82665ecc-4431-48fe-b255-{{Gerrit|7e9d518be217}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:48 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:48 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:47 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: cd284562-e3b0-4504-9977-{{Gerrit|2b18032bbf3c}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:45 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 8fb2a646-16a5-4183-94a1-{{Gerrit|ee6bbbf029fa}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:44 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:44 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: d561fe45-1582-4cd8-bbc9-{{Gerrit|a356c39f3330}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 351c365c-0228-4907-a279-{{Gerrit|01795b1e62ce}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 3beec1dd-0132-4f5c-adce-{{Gerrit|5a7dc56060b6}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:30 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:29 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:28 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:28 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:26 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:24 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:20 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:16 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 353a8527-ad5e-4898-937c-{{Gerrit|303d16801e28}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: fc235146-f070-4723-9503-{{Gerrit|e20cbb877755}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: cdf712ac-c083-4a63-af83-{{Gerrit|514ca5cdc76d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:05 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:01 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:01 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 8f2d2d32-e99e-4e2c-9472-{{Gerrit|ca33b577d01f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 66ee7d87-cee1-4c8f-b5c3-{{Gerrit|5f58a6cea336}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:59 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:59 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:58 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:58 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:58 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 69035b38-9910-46f1-9582-{{Gerrit|3105c1ba4d16}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:57 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:57 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:52 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 5a94c9d1-c956-4b03-a92b-{{Gerrit|b112810906a4}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:51 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:51 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:50 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 216a9120-4b9a-4be3-99db-{{Gerrit|3fd5fdc25146}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:49 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:49 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 46fb1964-3cf4-47fd-ad79-{{Gerrit|475f33c8ed38}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: b04a4f8c-db82-4258-8608-{{Gerrit|2a78ec9193ed}}, use this to unset). - cookbook ran by andrew@buster
* 19:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 7ae7c258-ecb8-47d1-a991-{{Gerrit|3fa617e305bf}}, use this to unset). - cookbook ran by andrew@buster
* 19:45 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 377fb0e5-132e-4e7a-9079-{{Gerrit|37937c3d42bf}}, use this to unset). - cookbook ran by andrew@buster
* 19:44 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 6c4222ae-796e-4890-8820-{{Gerrit|9e7ce5438f2a}}, use this to unset). - cookbook ran by andrew@buster
* 19:43 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:43 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:38 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:34 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:34 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:31 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: eff98e2e-ae99-41d0-a68e-{{Gerrit|671d941fcf21}}, use this to unset). - cookbook ran by andrew@buster
* 19:30 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5310f1eb-0425-4f59-8fb5-{{Gerrit|1d9c6f6d2f23}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:29 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:25 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: d8ab4682-026a-4ddc-bfc5-{{Gerrit|48166bd9789a}}, use this to unset). - cookbook ran by andrew@buster
* 19:24 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:23 wm-bot2: Safe reboot of 'cloudvirt1048.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:23 wm-bot2: Unset cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: eb2f9d94-8ea5-48d5-a336-{{Gerrit|97dd407b1ce3}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Safe rebooting 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:18 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:14 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 305fe2d7-913e-4638-95c2-{{Gerrit|4e810dc6b2a3}}, use this to unset). - cookbook ran by andrew@buster
* 19:14 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5bb3a66b-9450-4230-8d09-{{Gerrit|d78473d72ba2}}, use this to unset). - cookbook ran by andrew@buster
* 19:13 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:13 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:07 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: b1e39b49-3a88-4e42-913c-{{Gerrit|9892eafad447}}, use this to unset). - cookbook ran by andrew@buster
* 19:06 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:05 andrewbogott: putting cloudvirt1049-1052 into 'ceph' pool, taking out of 'spare' pool. cloudvirt1053 will remain our only spare.
* 19:02 wm-bot2: Safe reboot of 'cloudvirt1049.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:02 wm-bot2: Unset cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:00 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:00 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:59 wm-bot2: Safe reboot of 'cloudvirt1051.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:59 wm-bot2: Unset cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Drained 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Set cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance (downtime id: 8f24ea75-e107-4a13-8bd6-{{Gerrit|950b941b8e03}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Draining 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Safe rebooting 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:57 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:57 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:56 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:56 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 5c3e8fa4-cd1e-43f4-a423-{{Gerrit|ba7fe2459d4e}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Drained 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Set cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance (downtime id: e3fe22d6-932c-4502-8838-{{Gerrit|82cdc4c8f1c6}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:54 wm-bot2: Draining 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:54 wm-bot2: Safe rebooting 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:53 wm-bot2: Safe reboot of 'cloudvirt1053.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:53 wm-bot2: Unset cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: fd824b92-b6ff-4631-a2d4-{{Gerrit|debb25d48223}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:51 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:49 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 52d5cd5b-0f56-453a-85f8-{{Gerrit|0b5e656ae7f9}}, use this to unset). - cookbook ran by andrew@buster
* 18:49 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:49 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: b92f66a3-a487-4515-b68b-{{Gerrit|1c257a57b893}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:48 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-19 ===
* 20:07 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 4bfca6cd-dfbc-4a79-9203-{{Gerrit|b432bfeee5c2}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:41 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:21 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 70479325-609b-4349-9094-{{Gerrit|739eb9b3bb30}}, use this to unset). - cookbook ran by andrew@buster
* 19:21 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-09-14 ===
* 16:57 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
* 11:01 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:58 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:57 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:57 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:09 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:09 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:06 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:06 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 09:46 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-09-13 ===
* 12:16 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:16 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:11 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:11 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:36 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-09-10 ===
* 15:37 andrewbogott: restarting nova-conductor service (and possibly others, in response to lots of unanswered rabbitmq messages)

=== 2022-09-08 ===
* 19:12 andrewbogott: restarting nginx on proxy-03.project-proxy.eqiad1.wikimedia.cloud

=== 2022-09-07 ===
* 10:18 wm-bot2: Added OSD cloudcephosd1032.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:14 wm-bot2: Finished rebooting node cloudcephosd1032.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Rebooting node cloudcephosd1032.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Adding OSD cloudcephosd1032.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Adding new OSDs ['cloudcephosd1032.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:41 dhinus: Temporarily removing cloudcephosd1030 from Ceph cluster (https://phabricator.wikimedia.org/T314870)

=== 2022-08-30 ===
* 14:59 andrewbogott: manually marking most eqiad1 cloud* servers down in icinga for [[phab:T296561|T296561]]
* 10:43 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:39 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:38 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:38 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:39 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:14 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:11 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-25 ===
* 15:14 wm-bot2: Added 1 new OSDs ['cloudcephosd1029.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 15:14 wm-bot2: Added OSD cloudcephosd1029.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 15:02 wm-bot2: Finished rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:59 wm-bot2: Rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:58 wm-bot2: Adding OSD cloudcephosd1029.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:58 wm-bot2: Adding new OSDs ['cloudcephosd1029.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-24 ===
* 22:07 andrewbogott: replaced cloudservices1003 with cloudservices1005 [[phab:T304888|T304888]]
* 10:45 wm-bot2: Added 1 new OSDs ['cloudcephosd1028.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:45 wm-bot2: Added OSD cloudcephosd1028.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Finished rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:34 wm-bot2: Rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:33 wm-bot2: Adding OSD cloudcephosd1028.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:33 wm-bot2: Adding new OSDs ['cloudcephosd1028.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-23 ===
* 13:46 wm-bot2: Added 1 new OSDs ['cloudcephosd1027.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:46 wm-bot2: Added OSD cloudcephosd1027.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:27 wm-bot2: Finished rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:24 wm-bot2: Rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:22 wm-bot2: Adding OSD cloudcephosd1027.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:22 wm-bot2: Adding new OSDs ['cloudcephosd1027.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-21 ===
* 21:12 andrewbogott: restarted neutron-dhcp-agent on cloudnet1003. it was claiming to be unable to contact Rabbit but seems happy after a restart

=== 2022-08-20 ===
* 07:39 dcaro_away: cloudvirt1023 is back up, VMs are starting to recover ([[phab:T315718|T315718]])
* 07:23 dcaro_away: cloudvirt1023 seems to have gotten some hardware issue from racadm lclog view &quot;System CPU Resetting.&quot;, rebooting and doing memory checks ([[phab:T315718|T315718]])

=== 2022-08-19 ===
* 17:06 taavi: [codfw1dev] restart mariadb on clouddb2002-dev to pick up certificate config changes [[phab:T310795|T310795]]

=== 2022-08-18 ===
* 13:25 wm-bot2: Added 1 new OSDs ['cloudcephosd1026.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:25 wm-bot2: Added OSD cloudcephosd1026.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:15 wm-bot2: Finished rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:12 wm-bot2: Rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:10 wm-bot2: Adding OSD cloudcephosd1026.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:10 wm-bot2: Adding new OSDs ['cloudcephosd1026.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 07:29 dcaro: Starting up all the osd daemons on cloudcephosd1025 ([[phab:T314870|T314870]])

=== 2022-08-17 ===
* 10:50 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:49 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:50 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:49 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:49 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:16 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:16 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:16 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-16 ===
* 22:39 andrewbogott: replacing the now-rebuilt cloudvirt1025 in 'ceph' aggregate and removing it from the 'maintenance' aggregate
* 17:41 andrewbogott: removing cloudvirt1025 from the 'ceph' aggregate and adding it to the 'maintenance' aggregate
* 17:40 andrewbogott: reimaging cloudvirt1025 after I accidentally deleted the hw raid
* 17:38 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1003@rbd#RBD --newhost cloudcontrol1005@rbd#RBD
* 17:37 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1004@rbd#RBD --newhost cloudcontrol1006@rbd#RBD
* 16:26 wm-bot2: Ceph cluster at eqiad1 set out of maintenance. - cookbook ran by dcaro@vulcanus
* 15:43 wm-bot2: Restarting the osd daemons from nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,c
* 15:42 wm-bot2: Finished restarting all the OSD daemons from the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
* 15:38 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:02 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 12:59 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus

=== 2022-08-14 ===
* 18:36 taavi: deleted the http keystone endpoints from the keystone service catalog

=== 2022-08-11 ===
* 13:57 andrewbogott: decommissioning cloudcontrol1003 + cloudcontrl1004. I backed up $home in case anyone needs their files.
* 08:42 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:42 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:42 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-10 ===
* 13:10 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-04 ===
* 17:16 taavi: deleted all scheduler_fanout_ rabbit queues in an attempt to fix scheduling
* 16:32 taavi: restart neutron-l3-agent to pick up rabbit config changes
* 15:12 andrewbogott: stopping rabbitmq on cloudcontrol1xxx
* 09:57 taavi: stop wikitech_run_jobs.timer on labweb1001/1002, hosts pending decom

=== 2022-08-03 ===
* 20:55 andrewbogott: root@tools-checker-04:~# systemctl restart uwsgi-toolschecker_cron.service
* 20:41 andrewbogott: restarting  neutron-l3-agent.service on cloudnet1003 and 1004. The agent was routing properly but had lost touch with rabbitmq

=== 2022-08-02 ===
* 14:07 andrewbogott: shutting down codfw1dev ceph cluster according to https://docs.mirantis.com/mcp/q4-18/mcp-operations-guide/scheduled-maintenance-power-outage/power-off-ceph-cluster.html
* 13:54 andrewbogott: shutting down basically all of codfw1dev to support pdu maintenance -- all the ceph OSDs will lose power so best to have everything stopped.

=== 2022-07-27 ===
* 19:32 andrewbogott: switching the openstack.eqiad1.wikimedia.cloud endpoint from cloudcontrol1004 to 1006, https://gerrit.wikimedia.org/r/c/operations/dns/+/817878/2/templates/wikimediacloud.org#54
* 16:33 andrewbogott: here is a test message in the admin channel

=== 2022-07-25 ===
* 13:43 andrewbogott: pooling cloudweb100[34] and depooling labweb100[12] for testing in prep for decomming labweb100[12]

=== 2022-07-22 ===
* 16:41 taavi: depool cloudweb1003/1004 since horizon seems to be having issues
* 16:22 taavi: pooling cloudweb1003/1004 now that grant issues are sorted

=== 2022-07-21 ===
* 18:26 andrewbogott: depooling cloudweb1003 and 1004 for wikitech, horizon, striker -- pending db grant changes
* 18:06 andrewbogott: pooling cloudweb1003 and 1004 for wikitech, horizon, striker

=== 2022-07-20 ===
* 18:02 dcaro: things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 ([[phab:T313400|T313400]])
* 17:45 bd808: `sudo service striker restart` on labweb1002
* 17:43 bd808: `sudo service striker restart` on labweb1001
* 17:10 dcaro: things seem stable, trying to bring up a fourth rabbit node, cloudcontrol1006 ([[phab:T313400|T313400]])
* 16:26 dcaro: things seem stable, trying to bring up a third, cloudcontrol1005 ([[phab:T313400|T313400]])
* 15:51 dcaro: things seem stable now with one rabbit node, trying to bring up a second ([[phab:T313400|T313400]])
* 14:16 dcaro: stopping rabbin on cloudcontrol1004, leaving only 1003 alive ([[phab:T313400|T313400]])
* 13:17 dcaro: restarting the whole rabbit cluster ([[phab:T313400|T313400]])

=== 2022-07-19 ===
* 16:30 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:30 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 16:26 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:18 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:18 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 16:14 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:01 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:01 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:57 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:57 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 3da2d4f6-5c5b-4a21-9a0b-{{Gerrit|2b010960ed6a}}, use this to unset). - cookbook ran by andrew@buster
* 15:56 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:49 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 8a10505a-107d-4e78-ba84-{{Gerrit|363a7dea8d69}}, use this to unset). - cookbook ran by andrew@buster
* 15:47 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 752aa58b-44d4-4340-b05d-{{Gerrit|911b14f2314d}}, use this to unset). - cookbook ran by andrew@buster
* 15:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: f89f0851-ce3f-4a92-899e-{{Gerrit|0d4638acc9c3}}, use this to unset). - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:45 andrewbogott: adding new hosts to the 'ceph' aggregate: cloudvirt1046, 1047, 1048
* 15:44 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:44 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:42 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:42 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:40 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:38 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:37 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:37 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:33 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:32 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 32d9fdb7-6c42-4cf4-9950-{{Gerrit|6e2442255161}}, use this to unset). - cookbook ran by andrew@buster
* 15:31 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:31 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:16 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 56ce1388-792c-442c-bb89-{{Gerrit|e4c3869b0acf}}, use this to unset). - cookbook ran by andrew@buster
* 15:15 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:15 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:14 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 3f84bf2e-24de-4ebe-8117-{{Gerrit|0f4a5de29d09}}, use this to unset). - cookbook ran by andrew@buster
* 15:14 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:14 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:13 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:13 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:12 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 5418e923-997c-49ae-b937-{{Gerrit|35fc0c3038eb}}, use this to unset). - cookbook ran by andrew@buster
* 15:12 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: c3dfdc2b-0725-4d01-87f2-{{Gerrit|440a5ac4694c}}, use this to unset). - cookbook ran by andrew@buster
* 15:11 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:09 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:04 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:04 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:00 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:54 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:54 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:50 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:46 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 49dbf7de-c58a-4b68-bc7b-{{Gerrit|f001a047f4c7}}, use this to unset). - cookbook ran by andrew@buster
* 14:46 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:46 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:44 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: a4e95168-39b5-452d-8fce-{{Gerrit|7875ae44a62b}}, use this to unset). - cookbook ran by andrew@buster
* 14:44 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:44 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:43 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: b46234ee-9900-4708-a815-{{Gerrit|4324379be48c}}, use this to unset). - cookbook ran by andrew@buster
* 14:43 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:43 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:42 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:39 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:39 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:38 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:38 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 12bc87a9-7602-4795-818c-{{Gerrit|f7151b1c9ead}}, use this to unset). - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:35 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: ba787a26-62db-4662-ade2-{{Gerrit|2e3061b7390d}}, use this to unset). - cookbook ran by andrew@buster
* 14:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:29 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 6f0e79c6-2d91-478e-92df-{{Gerrit|0f65de48212f}}, use this to unset). - cookbook ran by andrew@buster
* 14:29 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 7f577c0f-0182-498f-b198-{{Gerrit|307d51df8c3a}}, use this to unset). - cookbook ran by andrew@buster
* 14:28 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: bd1bff92-bfb8-483d-a719-{{Gerrit|13fbdf3d8b21}}, use this to unset). - cookbook ran by andrew@buster
* 14:27 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:27 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: c9cd7dbb-c5ad-4364-a8e0-{{Gerrit|afb52ef98683}}, use this to unset). - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 4deca8a4-f73f-4b42-b4be-{{Gerrit|acfe61421ed0}}, use this to unset). - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 6d0d58b2-8450-480d-94f2-{{Gerrit|93a257f25575}}, use this to unset). - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:13 dcaro: deleting all the leftover fullstack images (was due to max number of mysql connections reached)
* 04:51 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:47 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:46 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 432747a2-9a15-44d4-ba8a-{{Gerrit|4e7e87eb8b76}}, use this to unset). - cookbook ran by andrew@buster
* 04:45 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1c5adb8c-efcf-4036-bf22-{{Gerrit|0eef364ae399}}, use this to unset). - cookbook ran by andrew@buster
* 04:44 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:44 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:41 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 26feb897-9eb0-40b6-9525-{{Gerrit|4cf44f81638d}}, use this to unset). - cookbook ran by andrew@buster
* 04:41 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:40 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:40 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:39 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: cd2424d3-c9c9-4837-93b1-{{Gerrit|a6d8271e7873}}, use this to unset). - cookbook ran by andrew@buster
* 04:39 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:39 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1e85ec01-81c4-4664-b14f-{{Gerrit|b1cf6417988c}}, use this to unset). - cookbook ran by andrew@buster
* 04:38 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f498aaa4-fad3-42f8-92f4-{{Gerrit|48a98c108456}}, use this to unset). - cookbook ran by andrew@buster
* 04:37 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 0ed281f3-d9b2-4c0a-bf88-{{Gerrit|db42fe848b2f}}, use this to unset). - cookbook ran by andrew@buster
* 04:35 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f417dffc-fa1b-4e65-938a-{{Gerrit|49e54b6fac6d}}, use this to unset). - cookbook ran by andrew@buster
* 04:34 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:33 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 923926e0-7194-4e13-98c4-{{Gerrit|a4f703b2907b}}, use this to unset). - cookbook ran by andrew@buster
* 04:32 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:32 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:29 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: ca8f729b-92ad-45ec-b5da-{{Gerrit|55987b0fc9c2}}, use this to unset). - cookbook ran by andrew@buster
* 04:29 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:29 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: abe6992d-dfba-4e50-993d-{{Gerrit|de0a551a177a}}, use this to unset). - cookbook ran by andrew@buster
* 04:24 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:24 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: df91107d-bf34-49db-9756-{{Gerrit|44ad06162683}}, use this to unset). - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f45ec0b1-83c9-49e0-a3a8-{{Gerrit|897d5a48414f}}, use this to unset). - cookbook ran by andrew@buster
* 04:23 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:18 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 504ae503-bfad-41bd-9079-{{Gerrit|fb53a9c82a62}}, use this to unset). - cookbook ran by andrew@buster
* 04:17 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:17 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 8410faf9-667d-443d-bd17-{{Gerrit|bb3ca8e7f725}}, use this to unset). - cookbook ran by andrew@buster
* 04:15 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:11 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 40441a37-0dd7-44a8-960b-{{Gerrit|f74f98f53618}}, use this to unset). - cookbook ran by andrew@buster
* 04:10 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:10 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:08 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 517cff2b-a643-4714-a31f-{{Gerrit|6bbe0767656a}}, use this to unset). - cookbook ran by andrew@buster
* 04:08 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 50b63ad6-4d8b-4fcf-9c17-{{Gerrit|27a106b6ec52}}, use this to unset). - cookbook ran by andrew@buster
* 04:07 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: d912bdf5-54c7-490a-bdbe-{{Gerrit|ba9a23f07f4b}}, use this to unset). - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: e75f16d0-e42f-4d72-a3bd-{{Gerrit|be0cbc5f200a}}, use this to unset). - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 87f6c417-3703-4f73-9876-{{Gerrit|20f6767dc8e1}}, use this to unset). - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: be14f869-fdb0-4c38-84ff-{{Gerrit|3d1e00d227eb}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: feb93ea7-2c64-4831-b140-{{Gerrit|dad8311dbcef}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: c65a23e3-a809-4a75-88e4-{{Gerrit|29a8b351ecdb}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:00 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:00 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:57 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:56 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8c8f64ca-a4e8-47cf-a2ce-{{Gerrit|23059109fb6a}}, use this to unset). - cookbook ran by andrew@buster
* 03:55 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:55 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:54 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 9d0457e8-a861-46ce-ac35-{{Gerrit|53975a46018e}}, use this to unset). - cookbook ran by andrew@buster
* 03:53 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:53 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:40 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:38 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:30 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 851d3b34-68f7-4c57-920b-{{Gerrit|2a64a9ea573d}}, use this to unset). - cookbook ran by andrew@buster
* 03:29 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:29 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:17 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 1060ab82-d53a-4d48-8e15-{{Gerrit|3198cabcfc2f}}, use this to unset). - cookbook ran by andrew@buster
* 03:17 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:17 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: dde1dc09-44b5-42af-ac49-{{Gerrit|c58dbbae7cb4}}, use this to unset). - cookbook ran by andrew@buster
* 03:14 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: b59fa99b-8713-4e63-8c56-{{Gerrit|08bd71fdfbe3}}, use this to unset). - cookbook ran by andrew@buster
* 03:13 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:07 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 3547c843-6b15-4151-b452-{{Gerrit|6f618a886b7b}}, use this to unset). - cookbook ran by andrew@buster
* 03:07 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 6f5808d0-afb2-4222-ae8e-{{Gerrit|4d953d49cd63}}, use this to unset). - cookbook ran by andrew@buster
* 03:06 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: da738ba3-883f-4d76-bb33-{{Gerrit|7eb551de5fc2}}, use this to unset). - cookbook ran by andrew@buster
* 03:05 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:05 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:04 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:04 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:03 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f90ca4c1-78af-46bd-9262-{{Gerrit|0fa4edc2898e}}, use this to unset). - cookbook ran by andrew@buster
* 03:02 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:00 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:00 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 62383663-9ae6-4a15-a2a6-{{Gerrit|62be06adbc96}}, use this to unset). - cookbook ran by andrew@buster
* 02:59 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:59 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:59 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 6e0da344-49ad-451f-8b8d-{{Gerrit|ea8f853a6f89}}, use this to unset). - cookbook ran by andrew@buster
* 02:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:52 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: c9ec5b7c-aa9f-4afd-a417-{{Gerrit|54c52c07ca47}}, use this to unset). - cookbook ran by andrew@buster
* 02:51 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:51 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:48 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 5509c648-58f6-49fd-90e8-{{Gerrit|ac9aa66e8b04}}, use this to unset). - cookbook ran by andrew@buster
* 02:48 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:48 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: af4ba9ad-e765-4087-86aa-{{Gerrit|9111c0821175}}, use this to unset). - cookbook ran by andrew@buster
* 02:46 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:45 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:45 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: 6da98123-8838-4d2e-8659-{{Gerrit|d769bfa292fe}}, use this to unset). - cookbook ran by andrew@buster
* 02:44 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:44 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:43 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 02:43 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:42 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:42 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 91860079-8fff-40db-9de7-{{Gerrit|17741d528ad6}}, use this to unset). - cookbook ran by andrew@buster
* 02:41 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:41 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:39 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: a6a0f414-d305-4a09-87d2-{{Gerrit|066eec83642b}}, use this to unset). - cookbook ran by andrew@buster
* 02:39 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 6000898c-4e5e-49c8-8841-{{Gerrit|3262d6da0366}}, use this to unset). - cookbook ran by andrew@buster
* 02:37 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:37 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 24a8b50e-5c04-40e6-a732-{{Gerrit|f0232b83e240}}, use this to unset). - cookbook ran by andrew@buster
* 02:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8d259aca-1803-4346-b5a4-{{Gerrit|a447673b3537}}, use this to unset). - cookbook ran by andrew@buster
* 02:34 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:33 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 704aca7f-0069-4a4a-b8de-{{Gerrit|1aba7b9ca7ef}}, use this to unset). - cookbook ran by andrew@buster
* 02:32 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04c0fba3-e2cb-40b4-abfd-{{Gerrit|d24165c9bd55}}, use this to unset). - cookbook ran by andrew@buster
* 02:32 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 25e5e0c8-5c27-460e-9b44-{{Gerrit|1cbd1c2fb551}}, use this to unset). - cookbook ran by andrew@buster
* 02:28 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:25 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5e62968a-0bb6-43b1-ae5e-{{Gerrit|e352442ef976}}, use this to unset). - cookbook ran by andrew@buster
* 02:24 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:24 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:21 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5b1a8e8f-8699-433d-b764-{{Gerrit|197862dc5cf1}}, use this to unset). - cookbook ran by andrew@buster
* 02:20 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:20 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:15 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:15 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:13 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 299c466e-602b-4a8e-bb37-{{Gerrit|156553dbe426}}, use this to unset). - cookbook ran by andrew@buster
* 02:13 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:13 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:12 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:12 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:03 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6a64d619-c484-411f-ad9d-{{Gerrit|89d43b063737}}, use this to unset). - cookbook ran by andrew@buster
* 02:02 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:02 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 301309a2-a8b5-4698-98d6-{{Gerrit|bb4aa0a75e45}}, use this to unset). - cookbook ran by andrew@buster
* 00:46 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: b2f8c6a2-7224-4fb1-8e12-{{Gerrit|dff6dda02380}}, use this to unset). - cookbook ran by andrew@buster
* 00:46 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:40 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:40 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:38 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: c0f62e0e-7865-4e77-a238-{{Gerrit|d707ceed53a8}}, use this to unset). - cookbook ran by andrew@buster
* 00:38 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:38 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:36 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 96a368d5-232f-4187-b0bb-{{Gerrit|5923a50fff71}}, use this to unset). - cookbook ran by andrew@buster
* 00:36 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:36 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:35 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 31b6bfcb-81cb-4d89-b977-{{Gerrit|190899ea2e64}}, use this to unset). - cookbook ran by andrew@buster
* 00:35 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:35 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:28 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 50f763d4-36ea-4241-8e71-{{Gerrit|cc6aa20ccc9e}}, use this to unset). - cookbook ran by andrew@buster
* 00:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: cd3d17af-ee15-4a9f-8750-{{Gerrit|66b97a46ea12}}, use this to unset). - cookbook ran by andrew@buster
* 00:27 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 9b075166-d424-42c0-8e06-{{Gerrit|f74a47abb798}}, use this to unset). - cookbook ran by andrew@buster
* 00:21 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-07-18 ===
* 23:11 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: e0aab64f-d911-4d7e-9a97-{{Gerrit|c59d9868ceea}}, use this to unset). - cookbook ran by andrew@buster
* 23:11 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 447d9c7a-a00e-41db-93ed-{{Gerrit|6d2bfc66e5df}}, use this to unset). - cookbook ran by andrew@buster
* 23:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:01 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 46c2ddf2-50c3-4824-ba72-{{Gerrit|d7fb3fa4c5e0}}, use this to unset). - cookbook ran by andrew@buster
* 23:01 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6facdf4f-7bb8-42cd-8ac8-{{Gerrit|b4412fc4cf70}}, use this to unset). - cookbook ran by andrew@buster
* 23:00 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:52 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 23190573-fc0d-4872-8343-{{Gerrit|e35b84132028}}, use this to unset). - cookbook ran by andrew@buster
* 22:51 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:51 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:51 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 990313c4-5fa6-4de7-a8bf-{{Gerrit|b361f0c79e90}}, use this to unset). - cookbook ran by andrew@buster
* 22:50 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:50 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:48 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:48 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:42 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 59367d39-bd2f-47fd-becd-{{Gerrit|255bf823ff12}}, use this to unset). - cookbook ran by andrew@buster
* 22:42 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:42 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:40 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 38602815-d187-4455-9676-{{Gerrit|59607505bba7}}, use this to unset). - cookbook ran by andrew@buster
* 22:39 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 00f21e1b-d014-4bc5-995f-{{Gerrit|e25201fc77c4}}, use this to unset). - cookbook ran by andrew@buster
* 22:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:31 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: d05d07b2-8c60-46de-a89e-{{Gerrit|76987901901b}}, use this to unset). - cookbook ran by andrew@buster
* 22:31 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 47f6b9e6-ceab-4fb6-8f76-{{Gerrit|0de4341d0841}}, use this to unset). - cookbook ran by andrew@buster
* 22:31 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:31 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 599042a5-da25-4985-b502-{{Gerrit|328fe40a72d5}}, use this to unset). - cookbook ran by andrew@buster
* 22:29 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:29 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:29 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: a554d107-c14d-41f8-b180-{{Gerrit|093e2cf73e72}}, use this to unset). - cookbook ran by andrew@buster
* 22:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9fef70c7-6f7d-4b33-a9bc-{{Gerrit|8c39630dc182}}, use this to unset). - cookbook ran by andrew@buster
* 22:26 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:02 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 3744c0fa-084d-4198-98ca-{{Gerrit|66c7a5210f41}}, use this to unset). - cookbook ran by andrew@buster
* 22:02 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 01caf1e1-df64-474a-94d5-{{Gerrit|f37751e23d62}}, use this to unset). - cookbook ran by andrew@buster
* 22:01 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:59 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 3018c2fc-9cd5-45c8-a160-{{Gerrit|d5ad5728c8d5}}, use this to unset). - cookbook ran by andrew@buster
* 21:59 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:59 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: d760f916-32ea-4194-8974-{{Gerrit|f36064a9866c}}, use this to unset). - cookbook ran by andrew@buster
* 21:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:00 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: cb2a38f3-e9ca-4bf6-989c-{{Gerrit|a27be2b2d88c}}, use this to unset). - cookbook ran by andrew@buster
* 20:59 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:58 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 93fa9477-54b8-46e7-8508-{{Gerrit|a2d492528d1f}}, use this to unset). - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f6cf5b29-ce8b-426e-af1c-{{Gerrit|cab74e0c4a3a}}, use this to unset). - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: fe26e612-d76e-4120-b60a-{{Gerrit|4f300add224f}}, use this to unset). - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 2abd94be-b6d8-4042-a9ce-{{Gerrit|998e41d932f4}}, use this to unset). - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 10973c71-30d8-44bf-a310-{{Gerrit|d0c1af76d399}}, use this to unset). - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f8f1b214-ec76-4157-a8e6-{{Gerrit|6374e5d14608}}, use this to unset). - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:11 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 95dae613-2b04-472e-a8f9-{{Gerrit|e71c3fbb8d0e}}, use this to unset). - cookbook ran by andrew@buster
* 20:10 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: c804f4db-ea63-44e9-aff8-{{Gerrit|fea54c238345}}, use this to unset). - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 597a901d-4624-43de-b986-{{Gerrit|c4c6c2fcf80c}}, use this to unset). - cookbook ran by andrew@buster
* 20:03 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:03 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: fdceed0d-6fa0-4806-8fb6-{{Gerrit|3b321a5a1623}}, use this to unset). - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:37 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: b0904345-9aa3-4202-b992-{{Gerrit|78644141517c}}, use this to unset). - cookbook ran by andrew@buster
* 19:36 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:31 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:31 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-07-13 ===
* 14:48 bd808: Added Tavvi as member of #acl*wmcs-team

=== 2022-07-12 ===
* 12:04 wm-bot2: Ceph cluster at &lt;nowiki&gt;{&lt;/nowiki&gt;self.deployment&lt;nowiki&gt;}&lt;/nowiki&gt; set out of maintenance. - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Set the ceph cluster for codfw1dev in maintenance, alert silence ids: db32805f-a033-4e0e-8fc7-{{Gerrit|ed0c2e9f8be1}},f4e698f0-4b51-4b07-9ba8-{{Gerrit|0b296ca1c4fb}},39ad5325-44ed-44d1-bba5-{{Gerrit|91a85c7a8401}},a584a3c6-9dae-41e9-ac82-{{Gerrit|a91cd35af8fb}} - cookbook ran by dcaro@vulcanus
* 08:37 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2005-dev', 'cloudnet2006-dev'] - cookbook ran by dcaro@vulcanus
* 08:37 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:31 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2005-dev,cloudnet2006-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-08 ===
* 15:57 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-07-07 ===
* 07:17 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T312509|T312509]]) - cookbook ran by dcaro@vulcanus
* 07:12 wm-bot2: Rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T312509|T312509]]) - cookbook ran by dcaro@vulcanus

=== 2022-07-06 ===
* 17:50 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: ['8a5b9eee-48c0-474d-8277-faeb05a2ea61', '65aad0fc-d887-47a3-b20c-d1ed461a2411', '86b078ae-3a27-4063-8c7c-198a2fe0c172'] - cookbook ran by dcaro@vulcanus

=== 2022-07-04 ===
* 13:27 wm-bot2: Rebooting cloudgw host cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot2: Rebooted cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Rebooting cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Rebooting all the cloudgw nodes from the eqiad1 deployment: cloudgw1001.eqiad.wmnet,cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:13 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 13:12 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:09 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:56 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:52 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:56 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 10:56 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:51 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:51 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:47 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:47 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:41 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 09:07 wm-bot2: Rebooting cloudnet host cloudnet1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:06 wm-bot2: Rebooted cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot2: Rebooting cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot2: Rebooting all the cloudnet nodes cloudnet1003,cloudnet1004 - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:53 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:53 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:47 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:47 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:55 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:44 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 07:44 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:36 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:36 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:09 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:05 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:04 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-03 ===
* 21:27 andrewbogott: rebuilding rabbit cluster in codfw1dev to get rid of some queues so unresponsive that they can't otherwise be deleted

=== 2022-07-02 ===
* 11:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-01 ===
* 15:35 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:30 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:29 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:25 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:42 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 14:42 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:32 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:02 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:58 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:58 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:54 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:52 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:37 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:34 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:30 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-06-30 ===
* 18:17 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 18:13 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:45 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 06:52 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-06-29 ===
* 08:45 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 08:41 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-06-28 ===
* 13:03 taavi: grant the tools project access to the g3.cores16.ram64.disk20.10xiops flavor [[phab:T301949|T301949]]

=== 2022-06-22 ===
* 16:48 taavi: restart designate-*.service on both cloudservices nodes
* 16:44 taavi: restart nova-conductor on all the cloudcontrol nodes
* 12:50 andrewbogott: rebooting each eqiad1 cloudcontrol node in hopes of getting a baseline re: openstack instability

=== 2022-06-21 ===
* 04:48 andrewbogott: stopping nova-fullstack agent on cloudcontrol1003; it's going to page us otherwise and we're all AFK tomorrow
* 04:02 andrewbogott: restarting rabbitmq on cloudcontrol100x (one at a time)

=== 2022-06-17 ===
* 17:53 andrewbogott: switching to a new python-based health check for galera and haproxy. This may make things more stable, or it may not. [[phab:T310664|T310664]]
* 06:15 taavi: restart neutron-linuxbridge-agent on cloudvirt1046

=== 2022-06-15 ===
* 11:34 taavi: restart neutron-linuxbridge-agent on cloudvirt1022

=== 2022-06-14 ===
* 16:26 wm-bot2: OSDs (['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloudcephosd1021', 'cl
* 14:38 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudceph
* 12:55 wm-bot2: OSDs (['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 12:44 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus

=== 2022-06-13 ===
* 11:14 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:07 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:05 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:04 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:03 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 09:15 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus

=== 2022-06-06 ===
* 13:21 andrewbogott: restarting mysql/galera on cloudcontrol100x in an attempt to stabilize some flapping there

=== 2022-06-02 ===
* 07:51 taavi: restart neutron-linuxbridge-agent.service on cloudvirt1034 [[phab:T309732|T309732]]
* 00:37 andrewbogott: updated nameservers for codfw1dev instances via 'openstack subnet set --dns-nameserver etc.'

=== 2022-06-01 ===
* 17:11 andrewbogott: restarting designate services in cloudservices1xxx hosts
* 16:37 taavi: root@cloudcontrol1005:~# cinder reset-state --state available 491066a4-16a9-4ce4-a9f6-{{Gerrit|182660616d77}} for [[phab:T309659|T309659]]

=== 2022-05-30 ===
* 17:43 andrewbogott: restarting neutron-rpc and neutron-api services on cloudcontrol1xxx

=== 2022-05-29 ===
* 14:55 andrewbogott: restarting nova services on all eqiad1 cloudcontrol nodes to recover from rabbit breakage
* 14:15 andrewbogott: restarting rabbitmq on all eqiad1 cloudcontrol nodes (one at a time)

=== 2022-05-25 ===
* 20:01 balloons: clean up cinder backup a bit, restart service due to network outage
* 20:01 balloons: cloudvirt1029 restarted nova due to network outage

=== 2022-05-19 ===
* 15:21 andrewbogott: resetting password for the 'troveguest' rabbitmq user. I think I may have broken this during a recent rebuild of the rabbitmq cluster

=== 2022-05-18 ===
* 15:42 andrewbogott: updated the 'debian-11.0-bullseye' glance image with a fresh build

=== 2022-05-14 ===
* 11:33 taavi: deleted projects 'ores' and 'ores-staging' [[phab:T308102|T308102]]

=== 2022-05-13 ===
* 06:20 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 06:20 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:16 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:16 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:15 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:15 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:11 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 06:11 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:10 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:10 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:09 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:07 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:06 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:06 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:05 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:51 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:50 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:50 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:47 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:47 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:44 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:44 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:41 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:40 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:40 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:38 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:37 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:37 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:30 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:29 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:29 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:19 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:12 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:12 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:08 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:01 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:52 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:48 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:48 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:44 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:31 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:24 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:21 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:21 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:18 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:37 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:36 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:36 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:34 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:34 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:27 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:27 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:25 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:25 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:55 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:55 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:04 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:23 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 01:23 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:19 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:01 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:01 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:25 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 00:25 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:20 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:19 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:19 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:11 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 00:11 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:07 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-12 ===
* 23:55 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:55 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:23 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:23 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:20 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:17 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:17 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:13 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:57 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:56 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:56 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:55 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:55 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:54 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:54 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:53 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:52 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:52 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:51 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:44 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:44 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:42 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:36 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:36 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:33 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:33 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:30 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:22 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:22 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:21 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:21 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:11 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:11 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:08 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:08 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:07 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:07 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:05 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:04 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:04 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:00 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:55 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:55 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:53 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:50 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:48 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:48 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:44 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:44 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:43 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:43 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:31 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:23 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:22 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:22 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:11 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:07 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:00 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:58 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:35 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:35 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:06 andrewbogott: stopping nfs-server on labstore1004 in preparation for reboot
* 04:12 andrewbogott: rebooting primary bastion (bastion-eqiad1-03.bastion.eqiad1.wikimedia.cloud) in hopes of resolving a problem with ssh proxying

=== 2022-05-11 ===
* 18:48 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:39 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:38 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:38 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:04 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:03 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:03 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 08:56 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 08:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 07:53 dcaro: test
* 04:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:43 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:43 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:39 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:23 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:09 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:08 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:08 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:04 andrewbogott: reset and recreated the rabbitmq cluster in eqiad1 to get around some broken queues.
* 03:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:25 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:25 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-10 ===
* 21:43 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:40 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:40 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:35 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:32 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:05 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:02 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:55 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:55 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:47 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:46 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:45 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:45 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:40 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:39 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:39 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:37 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:36 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:33 wm-bot: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 19:33 wm-bot: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:29 wm-bot: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:06 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:05 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:05 wm-bot: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:41 andrewbogott: rebooting cloud*-dev for [[phab:T307668|T307668]]
* 13:59 taavi: manually attached [[User:Dreamy Jazz]] to wikitech for a password reset (https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Manually_associate_an_LDAP_account_with_wikitech)

=== 2022-05-07 ===
* 01:33 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:32 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:30 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:21 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:18 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-03 ===
* 20:38 andrewbogott: upgrading clouddb2001-dev in place
* 18:18 taavi: updated 'puppet-enc' endpoints on the keystone catalog to use https and port 443

=== 2022-05-02 ===
* 16:56 dcaro: rebooting cloudmetrics1001

=== 2022-04-29 ===
* 14:22 andrewbogott: changing login.toolforge.org, bastion.toolforge.org, and dev.toolforge.org dns entries to refer to the new Buster bastions [[phab:T277653|T277653]] https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Timeline

=== 2022-04-27 ===
* 14:51 wm-bot: Finished rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloud
* 14:50 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:41 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:41 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot: Finished rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot: Rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot: Finished rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:01 wm-bot: Rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:01 wm-bot: Finished rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Finished rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:44 wm-bot: Rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:43 wm-bot: Finished rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:39 wm-bot: Rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:39 wm-bot: Finished rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:35 wm-bot: Rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:35 wm-bot: Finished rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Finished rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:26 wm-bot: Rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:26 wm-bot: Finished rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot: Rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot: Finished rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot: Rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot: Finished rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot: Rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot: Finished rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Finished rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:57 wm-bot: Rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:57 wm-bot: Finished rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Finished rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Finished rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:46 wm-bot: Rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:46 wm-bot: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024 - cookbo
* 12:15 wm-bot: Finished rebooting the nodes ['cloudcephmon1001', 'cloudcephmon1002', 'cloudcephmon1003'] - cookbook ran by dcaro@vulcanus
* 12:15 wm-bot: Finished rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:12 wm-bot: Rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:12 wm-bot: Finished rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:09 wm-bot: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:09 wm-bot: Finished rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:07 wm-bot: Rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:07 wm-bot: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:02 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:02 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:59 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:59 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 11:55 wm-bot: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] - cookbook ran by dcaro@vulcanus
* 11:55 wm-bot: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot: Rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:47 wm-bot: Rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:47 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:43 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:43 wm-bot: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev - cookbook ran by dcaro@vulcanus

=== 2022-04-26 ===
* 10:36 taavi: [codfw1dev] updated designate pool to 2004/2005-dev according to the instructions on https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS/Designate#Initial_designate/pdns_node_setup

=== 2022-04-22 ===
* 10:33 taavi: [codfw1dev] restart designate-sink on both new cloudservices host to fix rabbitmq connectivity

=== 2022-04-21 ===
* 05:38 andrewbogott: replaced cloudservices200[2,3] with cloudservices200[4,5]

=== 2022-04-19 ===
* 15:29 andrewbogott: stopping all VMs on cloudvirt1019, reimaging host

=== 2022-04-18 ===
* 15:23 andrewbogott: reimaging cloudvirt1020, leaving VMs in place
* 13:40 andrewbogott: shutting down many codfdfw1dev servers (including network infra!) for [[phab:T305469|T305469]]

=== 2022-04-14 ===
* 20:14 andrewbogott: restarting nova-api and nova-conductor services in a superstitious attempt to reduce open DB connections

=== 2022-04-13 ===
* 22:01 andrewbogott: restarting galera on cloudcontrols (one by one) to clear open connections

=== 2022-04-11 ===
* 15:59 taavi: created cloudinfra.wmcloud.org zone

=== 2022-04-09 ===
* 19:55 andrewbogott: reimaging cloudbackup1001-dev to bullseye
* 19:37 taavi: add 'puppet-enc' service &amp; endpoint to keystone [[phab:T274666|T274666]]
* 19:25 andrewbogott: reimaging cloudbackup1002-dev to bullseye

=== 2022-04-07 ===
* 12:51 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T305631|T305631]]) - cookbook ran by arturo@nostromo

=== 2022-04-06 ===
* 09:12 arturo: [codf1dev] installing python3-eventlet 0.30.2-5~bpo11+1 on all required servers (cloudvirt, cloudnet, cloudcontrol) ([[phab:T305157|T305157]])
* 08:45 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudvirt2003-dev ([[phab:T305157|T305157]])
* 08:42 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudcontrol servers ([[phab:T305157|T305157]])
* 08:24 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudvirt2003-dev ([[phab:T305157|T305157]])
* 08:20 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudcontrol servers ([[phab:T305157|T305157]])

=== 2022-03-30 ===
* 11:20 arturo: apply urpf strict filter to eqiad cloud-hosts vlan - [[phab:T285461|T285461]]

=== 2022-03-29 ===
* 10:02 dcaro: restarting keystone ([[phab:T304918|T304918]])

=== 2022-03-23 ===
* 22:53 wm-bot: Drained 'cloudvirt1045.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:38 wm-bot: Drained 'cloudvirt1044.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:12 wm-bot: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:12 wm-bot: Draining 'cloudvirt1045.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:08 wm-bot: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:07 wm-bot: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:06 wm-bot: Draining 'cloudvirt1044.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:06 wm-bot: Draining 'cloudvirt1043.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:54 wm-bot: Drained 'cloudvirt1042.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:19 wm-bot: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:19 wm-bot: Draining 'cloudvirt1042.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:12 wm-bot: Drained 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:12 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:09 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:07 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:04 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:55 wm-bot: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:54 wm-bot: Draining 'cloudvirt1041.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:30 wm-bot: Drained 'cloudvirt1039.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:14 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:14 wm-bot: Draining 'cloudvirt1039.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:44 wm-bot: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:43 wm-bot: Draining 'cloudvirt1038.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:19 wm-bot: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:18 wm-bot: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:13 wm-bot: Drained 'cloudvirt1036.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:02 wm-bot2: Testing wm-bot relay to #wikimedia-cloud-feed
* 17:55 wm-bot: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:54 wm-bot: Draining 'cloudvirt1036.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:04 wm-bot: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:03 wm-bot: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:03 wm-bot: Drained 'cloudvirt1034.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:51 wm-bot: Drained 'cloudvirt1033.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:37 wm-bot: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:37 wm-bot: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Draining 'cloudvirt1034.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 15:01 wm-bot: Drained 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 15:00 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:57 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:44 wm-bot: Drained 'cloudvirt1031.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:35 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:34 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:32 wm-bot: Drained 'cloudvirt1030.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:20 wm-bot: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:19 wm-bot: Draining 'cloudvirt1031.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:18 wm-bot: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:17 wm-bot: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:54 taavi: restart nova-fullstack on cloudcontrol1003 to pick up bastion ip change
* 13:43 wm-bot: Drained 'cloudvirt1029.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:23 wm-bot: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:22 wm-bot: Draining 'cloudvirt1029.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-22 ===
* 22:59 wm-bot: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:58 wm-bot: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-17 ===
* 01:09 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:53 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:52 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:52 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-15 ===
* 20:58 wm-bot: Drained 'cloudvirt1026.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:14 wm-bot: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 13:14 wm-bot: Unsetting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 10:32 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 10:30 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo

=== 2022-03-14 ===
* 21:24 wm-bot: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:59 wm-bot: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:58 wm-bot: Setting cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:58 wm-bot: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:02 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:59 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:59 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:16 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:13 wm-bot: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:56 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:55 wm-bot: Setting cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:55 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:53 wm-bot: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:52 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:51 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:51 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:50 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 11:48 dcaro: rebased cookbooks on latest master, make sure you pull before sending new patches

=== 2022-03-08 ===
* 18:29 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:29 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:29 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:23 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:21 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:21 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:18 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:17 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:17 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:27 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:27 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:18 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:15 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:15 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:47 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:47 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:08 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:07 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:07 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:11 arturo: [codfw1dev] rebooting cloudservices servers for [[phab:T303179|T303179]]
* 13:07 arturo: [codfw1dev] rebooting cloudvirt servers for [[phab:T303179|T303179]]
* 13:06 arturo: [codfw1dev] rebooting cloudnet servers for [[phab:T303179|T303179]]
* 12:55 arturo: [codfw1dev] rebooting cloudcontrol servers for [[phab:T303179|T303179]]

=== 2022-03-03 ===
* 08:49 taavi: deploying cloudmetrics grafana to grafana 8, [[phab:T282863|T282863]]

=== 2022-03-02 ===
* 09:06 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/701347

=== 2022-02-28 ===
* 15:30 dcaro: cleaning up leftover snapshots from failed backups of the maps volume ([[phab:T302720|T302720]])

=== 2022-02-24 ===
* 17:04 andrewbogott: upgrading eqiad1 and codfw1dev to mariadb 10.5.15+maria~bullseye via 'apt-get install libmariadb3:amd64 galera-4 mariadb-server'
* 15:42 dcaro: stopping and starting mariadb on cloudcontrol1003 ([[phab:T302146|T302146]])
* 10:37 arturo: [codfw1dev] briefly installed galera-4 (26.4.11+1bullseye) over (26.4.9-0+deb11u1) on cloudcontrol2001-dev and then downgrade again to verify package install ([[phab:T302482|T302482]])

=== 2022-02-23 ===
* 20:39 taavi: added domain-wide 'designateadmin' and 'observer' roles to project-proxy-dns-manager service account [[phab:T295246|T295246]]
* 17:40 andrewbogott: restarting lots of openstack services to try to clear up the mess that is [[phab:T236101|T236101]]
* 12:13 arturo: cleaning up cinder volume snapshots, aborrero@cloudcontrol1005:~$ for i in $(sudo wmcs-openstack volume snapshot list -f value -c ID) ; do sudo wmcs-openstack volume snapshot delete $i ; done ([[phab:T302382|T302382]])
* 10:14 arturo: cleaning up neutron agents for non-existent servers cloudvirt100[1-9].eqiad.wmnet,cloudvirt10[12-15].eqiad.wmnet
* 10:05 dcaro: Deleting stuck novafullstack servers, to let the service create new ones ([[phab:T302369|T302369]])
* 09:56 arturo: neutron agent-delete bad663b3-fd25-4393-a546-{{Gerrit|4b1b4bdec4db}} (Linux bridge agent {{!}} cloudvirtan1001)
* 09:56 arturo: neutron agent-delete 1071c198-ed57-4b5a-9439-{{Gerrit|30e66a31aa69}} (Linux bridge agent {{!}} cloudvirtan1005)
* 09:55 arturo: neutron agent-delete 2eeef198-8af7-4e5d-bd73-{{Gerrit|e14a2a8d2404}} (Linux bridge agent {{!}} cloudvirtan1004)
* 09:55 arturo: neutron agent-delete afe173eb-35ba-444a-9960-{{Gerrit|899629786d2f}} (Linux bridge agent {{!}} cloudvirtan1003)
* 09:54 arturo: neutron agent-delete afcb9b7f-c1a6-4ff4-9b10-{{Gerrit|92bfbe8d1a56}} (Linux bridge agent {{!}} cloudvirtan1002)
* 09:39 dcaro: restarting neutron-api cloudcontrol1003 to see if the agent status update starts working ([[phab:T302369|T302369]])
* 09:38 dcaro: restarting neutron-dhcp-agent on cloudnet1003 ([[phab:T302369|T302369]])

=== 2022-02-22 ===
* 22:10 andrewbogott: raising project 'maps' quota by two tb -- [[phab:T300160|T300160]]
* 09:24 arturo: restarting mariadb @ cloudcontrol1003 ([[phab:T302146|T302146]])
* 09:13 arturo: restarting mariadb @ cloudcontrol1004 ([[phab:T302146|T302146]])

=== 2022-02-18 ===
* 21:57 andrewbogott: leaving cloudcontrol1003 downtimed with disabled puppet for the weekend.  Everything there should be stable and fine save rabbit which needs an upgrade.
* 21:30 andrewbogott: rebooting cloudcontrol1003 because rabbit is freaking out
* 17:25 andrewbogott: in-place upgrade of cloudcontrol1004 to bullseye -- [[phab:T281276|T281276]]
* 12:34 arturo: manually install prometheus-openstack-exporter on cloudcontrol1005 ([[phab:T302050|T302050]])

=== 2022-02-17 ===
* 23:02 andrewbogott: in-place upgrade to Bullseye on cloudcontrol1005 [[phab:T281276|T281276]]

=== 2022-02-15 ===
* 14:15 taavi: [codfw1dev] added domain-wide 'designateadmin' and 'observer' roles to codfw1dev-proxy-dns-manager service account [[phab:T295246|T295246]]

=== 2022-02-04 ===
* 10:12 arturo: restart backup_vms service in cloudvirt1024 ([[phab:T300956|T300956]])

=== 2022-02-03 ===
* 08:21 taavi: cloudmetrics1004: manually added an empty line to /etc/prometheus/blackbox.yml to make /usr/local/bin/blackbox-exporter-assemble happy (clearing &quot;performing a change every puppet run&quot; alert)

=== 2022-02-02 ===
* 02:36 andrewbogott: restarting mariadb on cloudcontrol1004

=== 2022-01-31 ===
* 10:15 arturo: cloudcontrol1005:~$ sudo systemctl restart backup_glance_images.service (failed state, no logs, icinga alert)

=== 2022-01-29 ===
* 18:24 taavi: delete 2 puppet prefixes in a weird state [[phab:T299750|T299750]]

=== 2022-01-27 ===
* 13:24 arturo: cloudmetrics1004:~ $ sudo systemctl restart wmcs_monitoring_graphite_rsync.service ([[phab:T300138|T300138]])

=== 2022-01-26 ===
* 19:09 andrewbogott: bootstrapping a fresh galera node on cloudcontrol1004
* 18:57 andrewbogott: restarting mariadb on cloudcontrol1004

=== 2022-01-25 ===
* 10:49 arturo: made cloudmetrics1001/1002 primary/backup respectively ([[phab:T299744|T299744]], [[phab:T297814|T297814]], [[phab:T300011|T300011]])

=== 2022-01-19 ===
* 16:38 andrewbogott: moving all scratch mounts to scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud

=== 2022-01-05 ===
* 03:11 andrewbogott: 'cp /etc/apt/sources.list /etc/apt/sources.list.prepuppet' on all VMs. Backing up state before puppetizing sources.list with https://gerrit.wikimedia.org/r/c/operations/puppet/+/751498

=== 2022-01-04 ===
* 12:44 dcaro: increasing the size_limit for labs ldap servers

=== 2021-12-26 ===
* 16:55 majavah: run attachLdapUser.php on wikitech for developer account &quot;Karthiksripal&quot;

=== 2021-12-24 ===
* 22:51 majavah: ran the wikireplica dns script on s5 [[phab:T298303|T298303]]

=== 2021-12-23 ===
* 21:42 majavah: deployed horizon wmf-proxy-dashboard update to fix editing of existing proxies

=== 2021-12-21 ===
* 10:39 arturo: dropped egress NAT exceptions for WMF apt repos, [[phab:T298042|T298042]]

=== 2021-12-15 ===
* 12:44 dcaro: Downtiming cloudvirt-wdqs1001 as it has no VMs running until disk space is fixed ([[phab:T297454|T297454]])

=== 2021-12-14 ===
* 10:26 dcaro: Moved the nova cache (/var/lib/nova/instances/_base) and the canary image local data (/var/lib/nova/instance/&lt;canary_image_id&gt;) to the root disk on cloudvirt-wdqs1001 to temporary free some space ([[phab:T297454|T297454]])

=== 2021-12-13 ===
* 18:08 wm-bot: Drained 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:50 wm-bot: Set cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:49 wm-bot: Setting cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:49 wm-bot: Draining 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:44 wm-bot: Drained 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:30 wm-bot: Set cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:30 wm-bot: Setting cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:30 wm-bot: Draining 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:13 wm-bot: Drained 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
* 16:50 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:47 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:47 wm-bot: Draining 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
* 16:44 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:43 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse

=== 2021-12-03 ===
* 18:56 andrewbogott: maintain-views and maintain-meta-p on clouddb1013-1020
* 10:49 majavah: deleting dbbackups-dashboard project [[phab:T296992|T296992]]

=== 2021-12-02 ===
* 01:17 wm-bot: Drained 'cloudvirt1028.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:56 wm-bot: Set cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:56 wm-bot: Setting cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:56 wm-bot: Draining 'cloudvirt1028.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:50 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:50 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:50 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:28 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:03 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:02 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:02 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster

=== 2021-12-01 ===
* 23:59 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:59 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 23:54 andrewbogott: *correction* adding spare cloudvirts 1044 and 1045 to the 'ceph' pool in order to make space for future juggling around [[phab:T296790|T296790]] and [[phab:T296792|T296792]]
* 23:53 andrewbogott: adding spare cloudvirts 1044 and 1055 to the 'ceph' pool in order to make space for future juggling around [[phab:T296790|T296790]] and [[phab:T296792|T296792]]

=== 2021-11-28 ===
* 17:48 andrewbogott: moved cloudvirt1018 out of the 'localstorage' aggregate and into 'maintenance' for [[phab:T296592|T296592]].  It will need to be moved back after the raid is rebuilt.

=== 2021-11-21 ===
* 07:19 dcaro_away: restarting designate-sink with some extra logs in it ([[phab:T296144|T296144]])

=== 2021-11-17 ===
* 15:48 andrewbogott: upgrading mariadb packages on eqiad1 cloudcontrols
* 15:39 andrewbogott: sudo cumin  &quot;cloud*&quot; 'apt-get update -y --allow-releaseinfo-change'
* 15:26 andrewbogott: updated mariadb packages on codfw1dev cloudcontrols to 1:10.3.31-0+deb10u1

=== 2021-11-12 ===
* 13:31 arturo: restarting glance-api services to make sure they work with new ceph auth creds ([[phab:T293752|T293752]])

=== 2021-11-08 ===
* 21:50 andrewbogott: returned clouddb pools back to normal after maintain_views run: https://gerrit.wikimedia.org/r/c/operations/puppet/+/737505 [[phab:T216481|T216481]]
* 20:07 andrewbogott: depooling clouddb1013 for maintain_views attempt
* 10:54 arturo: [codfw1dev] create service account `srv-networktests` following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for [[phab:T294955|T294955]]
* 10:34 arturo: create service account `srv-networktests` following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for [[phab:T294955|T294955]]

=== 2021-11-05 ===
* 11:18 wm-bot: Added 1 new OSDs ['cloudcephosd1024.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:17 wm-bot: Added OSD cloudcephosd1024.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:15 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:12 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:12 wm-bot: Adding OSD cloudcephosd1024.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:12 wm-bot: Adding new OSDs ['cloudcephosd1024.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance

=== 2021-11-04 ===
* 16:39 wm-bot: Added 1 new OSDs ['cloudcephosd1023.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:39 wm-bot: Added OSD cloudcephosd1023.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:37 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:34 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:33 wm-bot: Adding OSD cloudcephosd1023.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:33 wm-bot: Adding new OSDs ['cloudcephosd1023.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:17 wm-bot: Added 1 new OSDs ['cloudcephosd1022.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:17 wm-bot: Added OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:16 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:13 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:12 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:12 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:00 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:00 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:26 wm-bot: Added 1 new OSDs ['cloudcephosd1021.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:26 wm-bot: Added OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:23 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:20 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:19 wm-bot: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:19 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:16 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance

=== 2021-11-03 ===
* 17:22 arturo: [codfw1dev] installing keepalived 2.1.5 from buster-backports on cloudgw2001-dev/2002-dev ([[phab:T294956|T294956]])
* 11:45 arturo: [codfw1dev] downgrade kernel on cloudgw2001-dev/2002-dev ([[phab:T294853|T294853]], [[phab:T291813|T291813]])

=== 2021-11-02 ===
* 10:54 arturo: rebooting cloudnet1004/1003 for [[phab:T291813|T291813]]
* 10:43 arturo: [codfw1dev] rebooting cloudgw200[12]-dev for [[phab:T291813|T291813]]

=== 2021-10-24 ===
* 00:47 andrewbogott: deploying a change so that openstack clients use tls endpoints: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732738

=== 2021-10-21 ===
* 10:19 arturo: drop firewall exception on core routers for wiki replicas legacy setup ([[phab:T293897|T293897]])
* 10:12 arturo: drop NAT exception for wiki replicas legacy setup ([[phab:T293897|T293897]])

=== 2021-10-20 ===
* 21:06 andrewbogott: creating cloudinfra-nfs project [[phab:T293936|T293936]]

=== 2021-10-18 ===
* 19:21 andrewbogott: also ticked the 'admin' box on wikitech for majavah [[phab:T292827|T292827]]
* 18:58 andrewbogott: granting majavah 'admin' role in the 'admin' project and also in the default domain.  [[phab:T292827|T292827]]

=== 2021-10-14 ===
* 12:28 arturo: [codfw1dev] add DB grants for cloudbackup2002.codfw.wmnet IP address to the cinder DB ([[phab:T292546|T292546]])

=== 2021-10-13 ===
* 10:46 arturo: updating python3-neutron across the fleet ([[phab:T292936|T292936]])

=== 2021-10-12 ===
* 09:06 dcaro: upgrading eqiad cloudnet hosts neutron packages ([[phab:T292936|T292936]])
* 08:57 dcaro: upgrading codfw cloudnet hosts neutron packages ([[phab:T292936|T292936]])

=== 2021-10-05 ===
* 09:39 arturo: [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) [[phab:T291257|T291257]]

=== 2021-09-30 ===
* 14:50 andrewbogott: sudo cumin  &quot;cloud*&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot; and sudo cumin  &quot;lab*&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot;  [[phab:T292202|T292202]]
* 14:43 andrewbogott: ran sudo cumin --force --timeout 500 -o json  &quot;A:all&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot; to get nslcd happy again [[phab:T292202|T292202]]

=== 2021-09-29 ===
* 09:41 arturo: [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured ([[phab:T291257|T291257]])

=== 2021-09-28 ===
* 16:23 bstorm: downtime for clouddb1020 to reduce re-pages in case this goes badly [[phab:T291963|T291963]]
* 16:21 bstorm: powering on clouddb1020 via remote console [[phab:T291963|T291963]]
* 15:58 bstorm: depooled clouddb1020 for repair [[phab:T291961|T291961]]
* 12:40 dcaro: Merged change on sssd for bullseye cloud hosts ([[phab:T291585|T291585]])
* 11:30 arturo: [codfw1dev] create floating IP 185.15.57.5 for manila-sharecontroller.cloudinfra-codfw1dev.codfw1dev.wmcloud.org ([[phab:T291257|T291257]])

=== 2021-09-27 ===
* 10:07 arturo: cloudcontrol1004 apparently healthy [[phab:T291446|T291446]]
* 09:25 arturo: rebooting cloudcontrol1004 for [[phab:T291446|T291446]]

=== 2021-09-24 ===
* 13:02 arturo: [codfw1dev] create VM manila-share-controller-01 on cloudinfra-codfw1dev
* 13:00 arturo: [codfw1dev] rebase labs/private.git on cloudinfra-puppetmaster-01, had merge conflict

=== 2021-09-21 ===
* 12:13 arturo: [codfw1dev] trying to create a manila service image ([[phab:T291257|T291257]])
* 11:45 arturo: [codfw1dev] created rabbitmq user ([[phab:T291257|T291257]])
* 11:32 arturo: [codfw1dev] populated manila DB &amp; created service endpoints ([[phab:T291257|T291257]])
* 11:06 arturo: [codfw1dev] give manila user admin role @ manila project ([[phab:T291257|T291257]])
* 11:06 arturo: [codfw1dev] created manila project ([[phab:T291257|T291257]])
* 10:57 arturo: [codfw1dev] created manila user @ labtestwikitech ([[phab:T291257|T291257]])
* 10:49 arturo: [codfw1dev] create manila database on cloudcontrol-dev nodes (galera) [[phab:T291257|T291257]]

=== 2021-09-20 ===
* 23:08 bstorm: ran `echo check &gt; /sys/block/md0/md/sync_action` on cloudcontrol1004 to check raid
* 22:48 andrewbogott: stopped puppet &amp; mariadb on cloudcontrol1004; it was flapping
* 22:44 andrewbogott: sudo touch /tmp/galera.disabled on cloudcontrol1004, the service seems troubled there
* 21:57 andrewbogott: moving cloudvirt1043 into the 'nfs' aggregate for [[phab:T291405|T291405]]

=== 2021-09-17 ===
* 11:35 arturo: [codfw1dev] install manila on cloudcontrol2001-dev ([[phab:T291257|T291257]])

=== 2021-09-16 ===
* 15:56 bstorm: removing downtime for labstore1005 so we'll know if it has another issue [[phab:T290318|T290318]]

=== 2021-09-09 ===
* 22:03 bstorm: restarted the prometheus-mysqld-exporter@s1 service as it was not working [[phab:T290630|T290630]]
* 03:15 bstorm: resetting swap on clouddb1017 [[phab:T290630|T290630]]
* 03:08 andrewbogott: stopping maintain-dbusers on labstore1004 for help diagnosing [[phab:T290630|T290630]]

=== 2021-09-03 ===
* 15:34 bstorm: rebooting labstore1005 to disconnect the drives from labstore1004 [[phab:T290318|T290318]]
* 15:24 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2002 [[phab:T290318|T290318]]
* 15:20 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2001 [[phab:T290318|T290318]]

=== 2021-08-30 ===
* 16:16 wm-bot: Added 1 new OSDs ['cloudcephosd1018.eqiad.wmnet'] - cookbook ran by andrew@buster
* 16:16 wm-bot: Added OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:10 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:07 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:07 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:07 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster

=== 2021-08-27 ===
* 18:57 andrewbogott: raising toolsbeta ram/core/instances quotas so majavah can experiment with bullseye

=== 2021-08-25 ===
* 14:45 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 14:42 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 14:42 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 14:42 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 14:41 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster

=== 2021-08-19 ===
* 17:39 bstorm: restarting glance image backup to try and clear the page

=== 2021-08-18 ===
* 16:21 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:21 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:21 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:17 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:16 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:15 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 14:47 andrewbogott: adding clouvirt1038 to the ceph aggregate, removing from the maintenance aggregate [[phab:T276922|T276922]]

=== 2021-08-17 ===
* 15:11 andrewbogott: rebooting cloudcephosd1008 to force raid rebuild -- [[phab:T287838|T287838]]

=== 2021-08-11 ===
* 13:51 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:48 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:47 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 13:47 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-08-10 ===
* 15:15 andrewbogott: restarting all designate services in eqiad1
* 15:04 andrewbogott: restarting designate-sink in eqiad1; it's complaining about rabbit but I don't want to restart rabbit yet

=== 2021-08-05 ===
* 09:37 dcaro: Taking one osd daemon down ot codfw cluster ([[phab:T288203|T288203]])

=== 2021-08-04 ===
* 19:20 bd808: Running deleteBatch.php on cloudweb2001-dev to remove legacy Heira: pages from labtestwiki

=== 2021-08-03 ===
* 17:40 bstorm: rerunning the glance backup script after failure

=== 2021-07-31 ===
* 00:10 andrewbogott: &quot;systemctl reset-failed cloud-init.service&quot; on all VMs for [[phab:T287309|T287309]]
* 00:08 andrewbogott: &quot;systemctl reset-failed cloud-final.service&quot; on all VMs for [[phab:T287309|T287309]]

=== 2021-07-27 ===
* 21:32 andrewbogott: putting cloudvirt1012 back into service [[phab:T286748|T286748]]
* 20:52 andrewbogott: draining VMs off of cloudvirt1012 so we can replace the battery for [[phab:T286748|T286748]]
* 15:15 andrewbogott: &quot;rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list&quot; cloud-wide

=== 2021-07-23 ===
* 15:22 bstorm: update wikireplicas-dns for s7 fix for web replicas

=== 2021-07-20 ===
* 17:07 andrewbogott: reloading haproxy on dbproxy1018 for [[phab:T286598|T286598]]
* 15:45 arturo: failback from labstore1006 to labstore1007 (dumps NFS) https://gerrit.wikimedia.org/r/c/operations/puppet/+/705417
* 00:10 bstorm: restarting nova-api on cloudcontrol1003 to try and recover whatever it's doing with designate_floating_ip_ptr_records_updater

=== 2021-07-19 ===
* 22:05 bstorm: set downtime scheduled for tomorrow from 1300 to 1600 UTC for cloudstore1008 and 1009 [[phab:T286599|T286599]]
* 20:40 andrewbogott: reloading haproxy on dbproxy1018 for [[phab:T286598|T286598]]
* 13:50 andrewbogott: upgrading mariadb to 10.3.29 on all cloudcontrols

=== 2021-07-16 ===
* 09:55 dcaro: checking HP raid issues on coludvirt1012 ([[phab:T286766|T286766]])

=== 2021-07-14 ===
* 21:08 andrewbogott: restarting lots of openstack services while trying to resolve [[phab:T286675|T286675]]
* 12:17 dcaro: doing ceph outage tests on codfw1 (fyi)

=== 2021-07-13 ===
* 10:57 dcaro: enabled autoscaling on codfw1 ceph cluster, setting a minimum of pgs on codfw1dev-compute to 128

=== 2021-07-02 ===
* 10:12 wm-bot: The cluster is not rebalance after adding the new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:12 wm-bot: Added 2 new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:12 wm-bot: Added OSD cloudcephosd1020.eqiad.wmnet... (2/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:10 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding OSD cloudcephosd1020.eqiad.wmnet... (2/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Added OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:02 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:02 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:01 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-07-01 ===
* 16:27 bstorm: failed over cloudstore1009 to cloudstore1008 [[phab:T224747|T224747]]
* 16:18 bstorm: downtimed cloudstore1008 and cloudstore1009 to fail over [[phab:T224747|T224747]]
* 14:25 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (2/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot: Added OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:20 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:20 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Added new OSD node cloudcephosd1016.eqiad.wmnet ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 11:13 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:58 dcaro: rebooting cloudcephosd1016 ([[phab:T285858|T285858]])
* 10:47 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:41 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-30 ===
* 21:48 bstorm: downtimed space alerts for scratch on cloudstore1008 until after the migration

=== 2021-06-25 ===
* 15:28 andrewbogott: restarting openstack services on cloudcontrol1005
* 09:16 arturo: icinga downtime cloudcontrols for 2h
* 08:20 dcaro: restarting rabbitmq on cloudcontrol100&lt;nowiki&gt;{&lt;/nowiki&gt;3,4&lt;nowiki&gt;}&lt;/nowiki&gt;

=== 2021-06-21 ===
* 13:54 dcaro: puppet fix merged and deployed, servers are back to normal
* 13:20 dcaro: merged broken puppet patch, downtimed all cloudvirts for 2h while fixing (nothing big, just added a bad systemd timer)

=== 2021-06-20 ===
* 22:21 andrewbogott: clearing admin-monitoring VMs; puppet has been failing lately due to a full drive on the puppetmaster

=== 2021-06-15 ===
* 01:18 bstorm: running a modified version of the prometheus dir size cron in screen [[phab:T284964|T284964]]

=== 2021-06-14 ===
* 10:13 dcaro: setting ssd to debug mode on tools-sgeexec-0917 ([[phab:T284130|T284130]])

=== 2021-06-10 ===
* 10:58 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:58 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:55 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:55 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:52 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:52 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:49 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:49 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:48 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:48 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:45 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:45 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:39 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:38 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:35 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:35 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:32 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:32 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:29 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:26 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:26 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:24 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:24 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-09 ===
* 17:33 arturo: removed icinga downtime for cloudmetrics1002 -- to see if hardware is healthy ([[phab:T281881|T281881]])
* 13:30 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:30 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-08 ===
* 23:19 bd808: Downtimed cloudmetrics1002 in icinga until 2021-06-30 23:59:01 ([[phab:T281881|T281881]])
* 21:08 bstorm: downtiming grafana-labs for maintenance
* 16:28 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:27 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:24 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:24 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:22 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:21 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:18 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:17 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

=== 2021-06-07 ===
* 14:27 andrewbogott: moving cloudvirt1040 from 'maintenance' aggregate to 'ceph' aggregate  [[phab:T281399|T281399]]

=== 2021-06-01 ===
* 13:12 dcaro: Changed the ceph osd_memory_target on eqiad pool to 6Gi (we were reaching the limit, swapping at some points)
* 09:57 arturo: fix PTR record for 185.15.56.1 ([[phab:T284025|T284025]])
* 09:56 arturo: fix PTR record for 185.15.56.1 ([[phab:T248025|T248025]])

=== 2021-05-27 ===
* 14:58 wm-bot: Testing - cookbook ran by dcaro@vulcanus

=== 2021-05-26 ===
* 19:10 andrewbogott: reimaging cloudvirt1018 to support local VM storage
* 18:07 andrewbogott: draining cloudvirt1018, converting it to a local-storage  host like cloudvirt1019 and 1020 -- [[phab:T283296|T283296]]
* 14:36 dcaro: Enabled syslog logging for osd.55 on eqiad ceph cluster for testing ([[phab:T281247|T281247]])
* 14:36 dcaro: Enabled syslog logging on codfw ceph cluster (mon/osd/mgr) ([[phab:T281247|T281247]])
* 11:26 arturo: [codfw1dev] purge old kernel packages in cloudvirt200[12]-dev
* 11:03 arturo: created public flavor `g3.cores16.ram36.disk20` (even though it was requested as private in [[phab:T283293|T283293]], but may be useful for others)

=== 2021-05-25 ===
* 16:14 bd808: Closed #wikimedia-cloud-admin on f***node
* 16:11 bd808: Closed #wikimedia-cloud-feed on f***node
* 15:19 dcaro: rebooted cloudvirt1020, starting VMs ([[phab:T275893|T275893]])
* 15:13 dcaro: rebooting cloudvirt1020 ([[phab:T275893|T275893]])
* 14:42 dcaro: taking cloudvirt1020 out for maintenance (openstack wise) so no new VMs are scheduled on it ([[phab:T275893|T275893]])

=== 2021-05-24 ===
* 22:32 andrewbogott: changing the default ttl for eqiad1.wikimedia.cloud. from 3600 to 60; this should help us avoid madness when re-using hostnames.
* 11:20 arturo: created `g3.cores2.ram80.disk40.private` for the wmf-research-tools project, to allow resizing a 40G disk instance

=== 2021-05-22 ===
* 02:14 bstorm: downtiming SMART alerts on dumps server labstore1007 for the weekend because it has been flapping [[phab:T281045|T281045]]

=== 2021-05-13 ===
* 21:25 bstorm: converted the maps and scratch volumes on cloudstore1008 (standby) to drbd [[phab:T224747|T224747]]
* 15:45 bstorm: re-running wikireplicas-dns after refactor of config to make sure it doesn't change anything

=== 2021-05-12 ===
* 14:23 arturo: [codfw1dev] cleanup old unused agents (bgp, ovs)
* 11:37 arturo: [codfw1dev] replacing cloudnet2003-dev with cloudnet2004-dev ([[phab:T281381|T281381]])

=== 2021-05-11 ===
* 18:00 andrewbogott: adding 'trove' service project in advance of deploying trove in eqiad1
* 10:22 arturo: rebooted cloudgw1002 (active) thus causing a failover to cloudgw1001

=== 2021-05-09 ===
* 10:53 arturo: icinga-downtime cloudmetrics1002 for 3 months ([[phab:T275605|T275605]])

=== 2021-05-07 ===
* 13:51 andrewbogott: add inherited 'admin' right to novaadmin user throughout eqiad1.  I was trying to narrow down the rights here but lack of admin breaks some workflows, e.g. [[phab:T281894|T281894]] and [[phab:T282235|T282235]]

=== 2021-05-06 ===
* 15:31 arturo: about to migrating CloudVPS network to the cloudgw architecture [[phab:T270704|T270704]]
* 11:14 dcaro: restarting cinder-volume on the eqiad control nodes to refresh the ceph libraries ([[phab:T282109|T282109]])

=== 2021-05-05 ===
* 16:07 dcaro: disallowing insecure global ids on the eqiad ceph cluster ([[phab:T280641|T280641]])
* 15:15 wm-bot: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:11 wm-bot: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:11 wm-bot: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:40 wm-bot: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:39 wm-bot: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:10 wm-bot: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 12:35 wm-bot: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:47 wm-bot: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:47 wm-bot: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:21 wm-bot: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:21 wm-bot: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:45 wm-bot: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:45 wm-bot: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:02 wm-bot: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus

=== 2021-05-04 ===
* 16:05 wm-bot: Safe reboot of 'cloudvirt1028.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:45 wm-bot: Safe rebooting 'cloudvirt1028.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:44 wm-bot: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:22 wm-bot: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:19 wm-bot: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:15 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 13:19 dcaro: rebooting cloudmetrics1002, got stuck again ([[phab:T275605|T275605]])
* 10:04 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:10 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:10 wm-bot: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:34 wm-bot: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:20 wm-bot: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:03 wm-bot: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus

=== 2021-05-03 ===
* 23:53 bstorm: running `maintain-dbusers harvest-replicas` on labstore1004 [[phab:T281287|T281287]]
* 23:51 bstorm: running `maintain-dbusers harvest-replicas` on labstore1004
* 16:34 wm-bot: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 16:29 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:41 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:41 wm-bot: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:31 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T280641|T280641]] - cookbook ran by dcaro@vulcanus)
* 10:23 wm-bot: (from a cookbook)
* 09:12 dcaro: draining and rebooting coludvirt1021 ([[phab:T280641|T280641]])
* 08:26 dcaro: draining and rebooting coludvirt1018 ([[phab:T280641|T280641]])

=== 2021-04-30 ===
* 11:16 dcaro: draining and rebooting coludvirt1017, last one today ([[phab:T280641|T280641]])
* 10:37 dcaro: draining coludvirt1016 for reboot ([[phab:T280641|T280641]])
* 09:48 dcaro: draining coludvirt1013 for reboot ([[phab:T280641|T280641]])

=== 2021-04-29 ===
* 15:11 dcaro: hard rebooting cloudmetrics1002, got hung again ([[phab:T275605|T275605]])
* 07:53 dcaro: Upgrading ceph libraries on cloudcontrol1005 to octopus ([[phab:T274566|T274566]])
* 07:51 dcaro: Upgrading ceph libraries on cloudcontrol1003 to octopus ([[phab:T274566|T274566]])
* 07:50 dcaro: Upgrading ceph libraries on cloudcontrol1004 to octopus ([[phab:T274566|T274566]])

=== 2021-04-28 ===
* 21:11 andrewbogott: cleaning up more references to deleted hypervisors with delete from services where topic='compute' and version != 53;
* 20:48 andrewbogott: cleaning up references to deleted hypervisors with mysql:root@localhost [nova_eqiad1]&gt; delete from compute_nodes where hypervisor_version != '5002000';
* 19:40 andrewbogott: putting cloudvirt1040 into the maintenance aggregate pending more info about [[phab:T281399|T281399]]
* 18:11 andrewbogott: adding cloudvirt1040, 1041 and 1042 to the 'ceph' host aggregate -- [[phab:T275081|T275081]]
* 11:06 dcaro: All ceph server side upgraded to Octopus! \o/ ([[phab:T280641|T280641]])
* 10:57 dcaro: Got a PG getting stuck on 'remapping' after the OSD came up, had to unset the norebalance and then set it again to get it unstuck ([[phab:T280641|T280641]])
* 10:34 dcaro: Slow/blocked opns from cloudcephmon03, &quot;osd_failure(failed timeout osd.32...&quot; (cloudcephosd1005), unset the cluster noout/norebalance and went away in a few secs, setting it again and continuing... ([[phab:T280641|T280641]])
* 09:03 dcaro: Waiting for slow heartbeats from osd.58(cloudcephosd1002) to recover... ([[phab:T280641|T280641]])
* 08:59 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) all from osd.58, currently on cloudcephosd1002 ([[phab:T280641|T280641]])
* 08:58 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) all from osd.58 ([[phab:T280641|T280641]])
* 08:58 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) ([[phab:T280641|T280641]])
* 08:21 dcaro: Upgrading all the ceph osds on eqiad ([[phab:T280641|T280641]])
* 08:21 dcaro: The clock skew seems intermittent, there's another task to follw it [[phab:T275860|T275860]] ([[phab:T280641|T280641]])
* 08:18 dcaro: All equiad ceph mons and mgrs upgraded ([[phab:T280641|T280641]])
* 08:18 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, cloudcephmon1001, they are back ([[phab:T280641|T280641]])
* 08:15 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, it went away, I'm guessing systemd-timesyncd fixed it ([[phab:T280641|T280641]])
* 08:14 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, looking ([[phab:T280641|T280641]])
* 07:58 dcaro: Upgrading ceph services on eqiad, starting with mons/managers ([[phab:T280641|T280641]])

=== 2021-04-27 ===
* 14:10 dcaro: codfw.openstack upgraded ceph libraries to 15.2.11 ([[phab:T280641|T280641]])
* 13:07 dcaro: codfw.openstack cloudvirt2002-dev done, taking cloudvirt2003-dev out to upgrade ceph libraries ([[phab:T280641|T280641]])
* 13:00 dcaro: codfw.openstack cloudvirt2001-dev back online, taking cloudvirt2002-dev out to upgrade ceph libraries ([[phab:T280641|T280641]])
* 10:51 dcaro: ceph.eqiad: cinder pool got it's pg_num increased to 1024, re-shuffle started ([[phab:T273783|T273783]])
* 10:48 dcaro: ceph.eqiad: Tweaked the target_size_ratio of all the pools, enabling autoscaler (it will increase cinder pool only) ([[phab:T273783|T273783]])
* 09:14 dcaro: manually force stopping the server puppetmaster-01 to unblock migration (in codfw1)
* 09:14 dcaro: manually force stopping the server puppetmaster-01 to unblock migration
* 08:59 dcaro: manually force stopping the server exploding-head on codfw, to try cold migration
* 08:47 dcaro: restarting nova-compute on cloudvirt2001-dev after upgrading ceph libraries to 15.2.11

=== 2021-04-26 ===
* 20:56 andrewbogott: deleting spurious 'codfw1dev' and 'codw1dev-4' regions in the dallas deployment; regions without endpoints break a bunch of things
* 09:45 dcaro: draining cloudvirt2001-dev with the new cookbooks ([[phab:T280641|T280641]])

=== 2021-04-23 ===
* 13:49 dcaro: testing the drain_cloudvirt cookbook on codfw1 openstack cluster, draining cloudvirt2001 ([[phab:T280641|T280641]])
* 11:12 dcaro: testing the drain_cloudvirt cookbook on codfw1 openstack cluster ([[phab:T280641|T280641]])
* 09:32 dcaro: finished upgrade of ceph cluster on codfw1 using exclusively cookbooks ([[phab:T280641|T280641]])
* 09:17 dcaro: testing the upgrade_osds cookbook on codfw1 ceph cluster ([[phab:T280641|T280641]])
* 08:17 dcaro: testing the upgrade_mons cookbook on codfw1 ceph cluster ([[phab:T280641|T280641]])

=== 2021-04-21 ===
* 17:59 dcaro: all monitors upgraded on codfw1 with one cookbook `cookbook --verbose -c ~/.config/spicerack/cookbook.yaml wmcs.ceph.upgrade_mons --monitor-node-fqdn cloudcephmon2002-dev.codfw.wmnet` ([[phab:T280641|T280641]])
* 17:47 dcaro: upgrading monitors and mrg nodes on codfw ceph cluster ([[phab:T280641|T280641]])
* 13:26 dcaro: testing ceph upgrade cookbook on cloudcephmon2002-dev ([[phab:T280641|T280641]])

=== 2021-04-20 ===
* 20:21 andrewbogott: reboot cloudservices1003
* 20:13 andrewbogott: reboot cloudservices1004

=== 2021-04-19 ===
* 08:40 dcaro: enabling puppet on labstore1004 after mysql restart ([[phab:T279657|T279657]])
* 08:09 dcaro: downtiming labstore1004 and stopping puppet for mysql restart ([[phab:T279657|T279657]])

=== 2021-04-14 ===
* 10:48 dcaro: Upgrade of codfw ceph to octopus 15.2.20 done, will run some performance tests now ([[phab:T274566|T274566]])
* 10:41 dcaro: Upgrade of codfw ceph to octopus 15.2.20, mgrs upgraded, osds next ([[phab:T274566|T274566]])
* 10:37 dcaro: Upgrade of codfw ceph to octopus 15.2.20, mons upgraded, mgrs next ([[phab:T274566|T274566]])
* 10:15 dcaro: starting the upgrade of codfw ceph to octopus 15.2.20 ([[phab:T274566|T274566]])
* 10:07 dcaro: Merged the ceph 15 (Octopus) repo deployment to codfw, only the repo, not the packages ([[phab:T274566|T274566]])

=== 2021-04-13 ===
* 16:42 dcaro: Ceph balancer got the cluster to eval 0.014916, that is 88-77% usage for compute pool, and 28-19% usage for the cinder one \o/ ([[phab:T274573|T274573]])
* 15:08 dcaro: Activating continuous upmap balancer, keeping a close eye ([[phab:T274573|T274573]])
* 15:03 dcaro: Executing a second pass, there's still movements to improve the eval of 0.030075 ([[phab:T274573|T274573]])
* 15:02 dcaro: First pass finished, improved eval to 0.030075 ([[phab:T274573|T274573]])
* 14:49 dcaro: Running the first_pass balancing plan on ceph eqiad, current eval 0.030622 ([[phab:T274573|T274573]])
* 14:43 dcaro: enabling ceph upmap pg balancer on equiad ([[phab:T274573|T274573]])
* 14:36 andrewbogott: upgrading codfw1dev to version Victoria, [[phab:T261137|T261137]]
* 13:11 andrewbogott: upgrading eqiad1 designate to version Victoria, [[phab:T261137|T261137]]
* 10:44 dcaro: enabled ceph upmap balancer on codfw ([[phab:T274573|T274573]],[[phab:T274573|T274573]])

=== 2021-04-07 ===
* 21:33 andrewbogott: upgrading codfw1dev designate to Victoria

=== 2021-04-04 ===
* 17:36 andrewbogott: upgrading eqiad1 designate to Ussuri

=== 2021-04-02 ===
* 14:12 andrewbogott: upgrading codfw1dev to OpenStack version Ussuri

=== 2021-04-01 ===
* 12:15 dcaro: Restoring the 4.9 kernel on cloudcephosd2003-dev and upgrading ([[phab:T274565|T274565]])
* 10:29 dcaro: Done restoring the 4.9 kernel on cloudcephosd2001-dev and upgrading, requires logging into console to boot from the older kernel before removing the newer one ([[phab:T274565|T274565]])
* 10:10 dcaro: Restoring the 4.9 kernel on cloudcephosd2001-dev and upgrading ([[phab:T274565|T274565]])

=== 2021-03-31 ===
* 08:47 dcaro: upgrading cinder on codfw cloudcontrol2* nodes ([[phab:T278845|T278845]])

=== 2021-03-30 ===
* 09:53 arturo: rebooting cloudnet1003 to cleanup conntrack table, it wouldn't cleanup by hand ...

=== 2021-03-28 ===
* 15:42 andrewbogott: updated debian-10.0-buster base image

=== 2021-03-27 ===
* 09:54 arturo: cleanup conntrack table in qrouter nents in cloudnet1003 (backup)

=== 2021-03-25 ===
* 19:03 andrewbogott: deleting all unused (per wmcs-imageusage) Jessie base images from Glance
* 17:15 andrewbogott: refreshing puppet compiler facts for tools project
* 10:31 dcaro: kernel upgrade on osds on codfw done, running performance tests ([[phab:T274565|T274565]])
* 10:24 dcaro: upgrading kernel on cloudcephosd2003-dev and reboot ([[phab:T274565|T274565]])
* 10:18 dcaro: upgrading kernel on cloudcephosd2002-dev and reboot ([[phab:T274565|T274565]])
* 10:08 dcaro: upgrading kernel on cloudcephmon2003-dev and reboot ([[phab:T274565|T274565]])

=== 2021-03-24 ===
* 09:19 dcaro: restarted wmcs-backup on cloudvirt1024 as it failed due to an image being removed while running ([[phab:T276892|T276892]])

=== 2021-03-23 ===
* 11:33 arturo: root@cloudcontrol1005:~# wmcs-novastats-dnsleaks --delete

=== 2021-03-22 ===
* 10:10 arturo: cleanup conntrack table in standby node: aborrero@cloudnet1003:~ $ sudo ip netns exec qrouter-d93771ba-2711-4f88-804a-{{Gerrit|8df6fd03978a}} conntrack -F

=== 2021-03-19 ===
* 17:18 bstorm: running `ALTER TABLE account MODIFY COLUMN type ENUM('user','tool','paws');` against the labsdbaccounts database on m5 [[phab:T276284|T276284]]
* 14:29 andrewbogott: switching admin-monitoring project to use an upstream debian image; I want to see how this affects performance
* 00:30 bstorm: downtimed labstore1004 to check some things in debug mode

=== 2021-03-17 ===
* 17:28 bstorm: restarted the backup-glance-images job to clear errors in systemd [[phab:T271782|T271782]]
* 17:16 andrewbogott: set default cinder quota for projects to 80Gb with &quot;update quota_classes set hard_limit=80 where resource='gigabytes';&quot; on database 'cinder'
* 16:58 andrewbogott: disabling all flavors with &gt;20Gb root storage with &quot;update flavors set disabled=1 where root_gb&gt;20;&quot; in nova_eqiad1_api

=== 2021-03-10 ===
* 16:51 arturo: rebooting cloudvirt1030 for [[phab:T275753|T275753]]
* 13:14 dcaro: starting manually the canary VM for cloudvirt1029 (nova start 349830f6-3b39-4a8c-ada4-{{Gerrit|a7439f65cffe}}) ([[phab:T275753|T275753]])
* 12:51 arturo: draining cloudvirt1030 for [[phab:T275753|T275753]]
* 12:47 arturo: rebooting cloudvirt1029 for [[phab:T275753|T275753]]
* 11:56 arturo: [codfw1dev] restart rabbitmq-server in all 3 cloudcontrol servers for [[phab:T276964|T276964]]
* 11:53 arturo: [codfw1dev] restart nova-conductor in all 3 cloudcontrol servers for [[phab:T276964|T276964]]
* 11:31 arturo: draining cloudvirt1029 for [[phab:T275753|T275753]]
* 11:29 arturo: rebooting cloudvirt1013 for [[phab:T275753|T275753]]
* 11:05 arturo: draining cloudvirt1013 for [[phab:T275753|T275753]]
* 11:00 arturo: rebooting cloudvirt1028 for [[phab:T275753|T275753]]
* 10:33 arturo: draining cloudvirt1028 for [[phab:T275753|T275753]]
* 10:29 arturo: rebooting cloudvirt1023 for [[phab:T275753|T275753]]
* 09:37 arturo: draining cloudvirt1023 for [[phab:T275753|T275753]]
* 09:07 arturo: [codfw1dev] reimaging cloudvirt2003-dev ([[phab:T276964|T276964]])

=== 2021-03-09 ===
* 16:27 arturo: rebooting cloudvirt1027 ([[phab:T275753|T275753]])
* 13:39 arturo: draining cloudvrit1027 for [[phab:T275753|T275753]]
* 13:35 arturo: icinga-downtime cloudvirt1038 for 30 days for [[phab:T276922|T276922]]
* 13:21 arturo: add cloudvirt1039 to the ceph host aggregate (no longer a spare, we have cloudvirt1038 with HW failures)
* 12:52 arturo: cloudvirt1038 hard powerdown / powerup for [[phab:T276922|T276922]]
* 12:33 arturo: rebooting cloudvirt1038 ([[phab:T275753|T275753]])
* 10:58 arturo: draining cloudvirt1038 ([[phab:T275753|T275753]])
* 10:54 arturo: rebooting cloudvirt1037 ([[phab:T275753|T275753]])
* 09:59 arturo: draining cloudvirt1037 ([[phab:T275753|T275753]])
* 09:12 dcaro: restarted the wmcs-backup service on cloudvirt1024 to retry the backups (failed because a VM was removed in-between, [[phab:T276892|T276892]])

=== 2021-03-05 ===
* 21:40 andrewbogott: replacing 'observer' role with 'reader' role in eqiad1 [[phab:T276018|T276018]]
* 21:21 andrewbogott: replacing 'observer' role with 'reader' role in eqiad1
* 16:23 arturo: rebooting cloudvirt1036 for [[phab:T275753|T275753]]
* 12:30 arturo: draining cloudvirt1036 for [[phab:T275753|T275753]]
* 12:25 arturo: rebooting cloudvirt1035 for [[phab:T275753|T275753]]
* 10:49 arturo: rebooting cloudvirt1035 for [[phab:T275753|T275753]]
* 10:47 arturo: rebooting cloudvirt1034 for [[phab:T275753|T275753]]
* 10:26 arturo: draining cloudvirt1034 for [[phab:T275753|T275753]]
* 10:25 arturo: rebooting cloudvirt1033 for [[phab:T275753|T275753]]
* 09:18 arturo: draining cloudvirt1033 for [[phab:T275753|T275753]]

=== 2021-03-04 ===
* 18:36 andrewbogott: rebooting cloudmetrics1002; the console is hanging
* 16:59 arturo: rebooting cloudvirt1032 for [[phab:T275753|T275753]]
* 16:34 arturo: draining cloudvirt1032 for [[phab:T275753|T275753]]
* 16:33 arturo: rebooting cloudvirt1031 for [[phab:T275753|T275753]]
* 16:11 arturo: draining cloudvirt1031 for [[phab:T275753|T275753]]
* 16:09 arturo: rebooting cloudvirt1026 for [[phab:T275753|T275753]]
* 15:57 arturo: draining cloudvirt1026 for [[phab:T275753|T275753]]
* 15:55 arturo: rebooting cloudvirt1025 for [[phab:T275753|T275753]]
* 15:41 arturo: draining cloudvirt1025 for [[phab:T275753|T275753]]
* 15:12 arturo: rebooting cloudvirt1024 for [[phab:T275753|T275753]]
* 11:29 arturo: draining cloudvirt1024 for [[phab:T275753|T275753]]
* 11:24 dcaro: rebooted cloudvirt1022, re-adding to ceph and removing from maintenance host aggregate for [[phab:T275753|T275753]]
* 11:01 dcaro: rebooting cloudvirt1022 for [[phab:T275753|T275753]]
* 09:12 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]

=== 2021-03-03 ===
* 17:16 andrewbogott: restarting rabbitmq-server on cloudcontrol1003,1004,1005; trying to explain amqp errors in scheduler logs
* 16:03 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]
* 16:03 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]
* 16:00 arturo: move cloudvirt1013 into the 'toobusy' host aggregate, it has 221% cpu subscription and 82% MEM subscription
* 15:34 arturo: rebooting cloudvirt1021 for [[phab:T275753|T275753]]
* 14:31 arturo: draining cloudvirt1021 for [[phab:T275753|T275753]]
* 13:59 arturo: rebooting cloudvirt1018 for [[phab:T275753|T275753]]
* 13:28 arturo: draining cloudvirt1018 for [[phab:T275753|T275753]]
* 12:49 arturo: rebooting cloudvirt1017 for [[phab:T275753|T275753]]
* 12:22 arturo: draining cloudvirt1017 for [[phab:T275753|T275753]]
* 12:20 arturo: rebooting cloudvirt1016 for [[phab:T275753|T275753]]
* 12:01 arturo: draining cloudvirt1016 for [[phab:T275753|T275753]]
* 11:59 arturo: cloudvirt1014 now in the ceph host aggregate
* 11:58 arturo: rebooting cloudvirt1014 for [[phab:T275753|T275753]]
* 11:50 arturo: moved cloudvirt1023 away from the maintenance host aggregate, leave it in the ceph aggregate (was in the 2)
* 11:47 arturo: moved cloudvirt1014 to the 'maintenance' host aggregate, drain it for [[phab:T275753|T275753]]
* 10:01 arturo: icinga-downtime cloudnet1003 for 14 days bc potential alerting storm due to firmware issues ([[phab:T271058|T271058]])
* 10:01 arturo: rebooting again cloudnet1003 (no network failover) ([[phab:T271058|T271058]])
* 09:59 arturo: update firmware-bnx2x from 20190114-2 to 20200918-1~bpo10+1 on cloudnet1003 ([[phab:T271058|T271058]])
* 09:30 arturo: installing linux kernel 5.10.13-1~bpo10+1 in cloudnet1003 and rebooting it (network failover) ([[phab:T271058|T271058]])

=== 2021-03-02 ===
* 17:16 andrewbogott: rebooting cloudvirt1039 to see if I can trigger [[phab:T276208|T276208]]
* 16:10 arturo: [codfw1dev] restart nova-compute on cloudvirt2002-dev
* 11:59 arturo: moved cloudvirt1012 to 'maintenance' host aggregate. Drain it with `wmcs-drain-hypervisor` to reboot it for [[phab:T275753|T275753]]
* 11:59 arturo: cloudvirt1023 is affected by [[phab:T276208|T276208]] and cannot be rebooted. Put it back into the ceph hos aggregate
* 10:43 arturo: moved cloudvirt1013 cloudvirt1032 cloudvirt1037 back into the 'ceph' host aggregate
* 10:13 arturo: moved cloudvirt1023 to 'maintenance' host aggregate. Drain it with `wmcs-drain-hypervisor` to reboot it for [[phab:T275753|T275753]]

=== 2021-03-01 ===
* 20:12 andrewbogott: removing novaadmin from all projects save 'admin' for [[phab:T274385|T274385]]
* 19:51 andrewbogott: removing novaobserver from all projects save 'observer' for [[phab:T274385|T274385]]
* 19:50 andrewbogott: adding inherited domain-wide roles to novaadmin and novaobserver as per [[phab:T274385|T274385]]

=== 2021-02-28 ===
* 04:54 andrewbogott: restarted redis-server on tools-redis-1003 and tools-redis-1004 in an attempt to reduce replag, no real change detected

=== 2021-02-27 ===
* 00:33 andrewbogott: sudo cumin --timeout 500 &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i buster &amp;&amp; uname -r  {{!}} grep -v 4.19.0-14-amd64 &amp;&amp; reboot'
* 00:28 andrewbogott: sudo cumin --timeout 500 &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i buster &amp;&amp; uname -r  {{!}} grep -v 4.19.0-14-amd64 &amp;&amp; echo reboot'
* 00:09 andrewbogott: sudo cumin &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i stretch &amp;&amp; uname -r  {{!}} grep -v 4.19.0-0.bpo.14-amd64 &amp;&amp; reboot'

=== 2021-02-26 ===
* 14:58 dcaro: [eqiad] rebooting cloudcephosd1015 (last osd \o/) for kernel upgrade ([[phab:T275753|T275753]])
* 14:51 dcaro: [eqiad] rebooting cloudcephosd1014 for kernel upgrade ([[phab:T275753|T275753]])
* 14:44 dcaro: [eqiad] rebooting cloudcephosd1013 for kernel upgrade ([[phab:T275753|T275753]])
* 14:38 dcaro: [eqiad] rebooting cloudcephosd1012 for kernel upgrade ([[phab:T275753|T275753]])
* 14:31 dcaro: [eqiad] rebooting cloudcephosd1011 for kernel upgrade ([[phab:T275753|T275753]])
* 14:25 dcaro: [eqiad] rebooting cloudcephosd1010 for kernel upgrade ([[phab:T275753|T275753]])
* 14:17 dcaro: [eqiad] rebooting cloudcephosd1009 for kernel upgrade ([[phab:T275753|T275753]])
* 13:54 dcaro: [eqiad] downtimed alert1001 Ceph OSDs down alert until 18:00 GMT+1 as that is not under the host being rebooted ([[phab:T275753|T275753]])
* 13:51 dcaro: [eqiad] rebooting cloudcephosd1008 for kernel upgrade ([[phab:T275753|T275753]])
* 13:45 dcaro: [eqiad] rebooting cloudcephosd1007 for kernel upgrade ([[phab:T275753|T275753]])
* 13:38 dcaro: [eqiad] rebooting cloudcephosd1006 for kernel upgrade ([[phab:T275753|T275753]])
* 12:07 dcaro: [eqiad] rebooting cloudcephosd1005 for kernel upgrade ([[phab:T275753|T275753]])
* 12:00 arturo: rebooting cloudcontrol1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:42 arturo: rebooting cloudcontrol1004 for kernel upgrade ([[phab:T275753|T275753]])
* 11:41 dcaro: [eqiad] rebooting cloudcephosd1004 for kernel upgrade ([[phab:T275753|T275753]])
* 11:32 dcaro: [eqiad] rebooting cloudcephosd1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:30 arturo: rebooting cloudcontrol1005 for kernel upgrade ([[phab:T2|T2]]
* 11:26 dcaro: [eqiad] rebooting cloudcephosd1002 for kernel upgrade ([[phab:T275753|T275753]])
* 11:16 dcaro: [eqiad] rebooting cloudcephosd1001 for kernel upgrade ([[phab:T275753|T275753]])
* 11:11 dcaro: [eqiad] rebooting cloudcephmon1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:05 dcaro: [eqiad] rebooting cloudcephmon1002 for kernel upgrade ([[phab:T275753|T275753]])
* 10:59 dcaro: [eqiad] rebooting cloudcephmon1001 for kernel upgrade ([[phab:T275753|T275753]])
* 10:45 arturo: rebooting cloudvirt1039 into a new kernel ([[phab:T275753|T275753]]) --- spare
* 10:43 dcaro: [codfw1dev] rebooting cloudcephmon2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:38 dcaro: [codfw1dev] rebooting cloudcephmon2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:29 dcaro: [codfw1dev] rebooting cloudcephmon2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:24 arturo: [codfw1dev] purge old kernel packages on cloudvirt2003-dev to force boot into a new kernel ([[phab:T275753|T275753]])
* 10:11 arturo: [codfw1dev] manually creating /boot/grub/ on cloudvirt2003-dev to allow update-grub2 to run (so it can reboot into a new kernel) ([[phab:T275753|T275753]])
* 10:11 dcaro: [codfw1dev] rebooting cloudcephosd2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:05 dcaro: [codfw1dev] rebooting cloudcephosd2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:01 arturo: [codfw1dev] rebooting cloudvirt200X-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:59 arturo: [codfw1dev] rebooting cloudweb2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:53 arturo: [codfw1dev] rebooting cloudservices2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:51 arturo: [codfw1dev] rebooting cloudservices2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:45 arturo: [codfw1dev] rebooting cloudcontrol2004-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:44 arturo: [codfw1dev] rebooting cloudbackup[2001-2002].codfw.wmnet for kernel upgrade ([[phab:T275753|T275753]])
* 09:43 dcaro: [codfw1dev] rebooting cloudcephosd2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:41 arturo: [codfw1dev] rebooting cloudcontrol2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:33 arturo: [codfw1dev] rebooting cloudcontrol2001-dev for kernel upgrade ([[phab:T275753|T275753]])

=== 2021-02-25 ===
* 14:56 arturo: deployed wmcs-netns-events daemon to all cloudnet servers ([[phab:T275483|T275483]])

=== 2021-02-24 ===
* 11:07 arturo: force-reboot cloudmetrics1002, add icinga downtime for 2 hours. Investigating some server issue
* 00:17 bstorm: set  --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main stretch image in glance on eqiad1 [[phab:T275430|T275430]]

=== 2021-02-23 ===
* 22:43 bstorm: set  --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main buster image in glance on eqiad1 [[phab:T275430|T275430]]
* 20:36 andrewbogott: adding r/o access to the eqiad1-glance-images ceph pool for the client.eqiad1-compute for [[phab:T275430|T275430]]
* 10:49 arturo: rebooting clounet1004 into new kernel from buster-bpo ([[phab:T271058|T271058]])
* 10:49 arturo: installing linux-image-amd64 from buster-bpo 5.10.13-1~bpo10+1 in cloudnet1004 ([[phab:T271058|T271058]])

=== 2021-02-22 ===
* 17:15 bstorm: restarting nova-compute on cloudvirt1016 and cloudvirt1036 in case it helps [[phab:T275411|T275411]]
* 15:02 dcaro: Re-uploaded the debian buster 10.0 image from rbd to glance, that worked, re-spawning all the broken instances ([[phab:T275378|T275378]])
* 11:12 dcaro: Refreshing all the canary instances ([[phab:T275354|T275354]])

=== 2021-02-18 ===
* 14:50 arturo: rebooting cloudnet1004 for [[phab:T271058|T271058]]
* 10:25 dcaro: Rebooting cloudmetrics1001 to apply new kernel ([[phab:T275116|T275116]])
* 10:16 dcaro: Rebooting cloudmetrics1002 to apply new kernel ([[phab:T275116|T275116]])
* 10:14 dcaro: Upgrading grafana on cloudmetrics1002 ([[phab:T275116|T275116]])
* 10:12 dcaro: Upgrading grafana on cloudmetrics1001 ([[phab:T275116|T275116]])

=== 2021-02-17 ===
* 15:58 arturo: deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/664845 to cloudnet servers ([[phab:T268335|T268335]])

=== 2021-02-15 ===
* 16:25 arturo: [codfw1dev] rebooting all cloudgw200x-dev / cloudnet200x-dev servers ([[phab:T272963|T272963]])
* 15:45 arturo: [codfw1dev] drop subnet definition for cloud-instances-transport1-b-codfw ([[phab:T272963|T272963]])
* 15:45 arturo: [codfw1dev] connect virtual router cloudinstances2b-gw to vlan cloud-gw-transport-codfw (185.15.57.10) ([[phab:T272963|T272963]])

=== 2021-02-11 ===
* 12:01 arturo: [codfw1dev] drop instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` (was buster, cannot use it yet)
* 11:59 arturo: [codfw1dev] create instance `tools-codfw1dev-bastion-2` (stretch) in `tools-codfw1dev` to test stuff related to [[phab:T272397|T272397]]
* 11:45 arturo: [codfw1dev] create instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` to test stuff related to [[phab:T272397|T272397]]
* 11:42 arturo: [codfw1dev] drop `tools` project, create `tools-codfw1dev`
* 11:38 arturo: [codfw1dev] drop `coudinfra` project (we are using `cloudinfra-codfw1dev` there)
* 05:37 bstorm: downtimed cloudnet1004 for another week [[phab:T271058|T271058]]

=== 2021-02-09 ===
* 15:23 arturo: icinga-downtime for 2h everything *labs *cloud for openstack upgrades
* 11:14 dcaro: Merged the osd scheduler change for all osds, applying on all cloudcephosd* ([[phab:T273791|T273791]])

=== 2021-02-08 ===
* 18:50 bstorm: enabled puppet on cloudvirt1023 for now [[phab:T274144|T274144]]
* 18:44 bstorm: restarted the backup_vms.service on cloudvirt1027 [[phab:T274144|T274144]]
* 17:51 bstorm: deleted project pki [[phab:T273175|T273175]]

=== 2021-02-05 ===
* 10:59 arturo: icinga-downtime labstore1004 tools share space check for 1 week ([[phab:T272247|T272247]])
* 10:21 dcaro: This was affecting maps and several others, maps and project-proxy have been fixed ([[phab:T273956|T273956]])
* 09:19 dcaro: Some certs around the infra are expired ([[phab:T273956|T273956]])

=== 2021-02-04 ===
* 10:12 dcaro: Increasing the memory limit of osds in eqiad from 8589934592(8G) to 12884901888(12G) ([[phab:T273851|T273851]])

=== 2021-02-03 ===
* 09:59 dcaro: Doing a full vm backup on cloudvirt1024 with the new script ([[phab:T260692|T260692]])
* 01:50 bstorm: icinga-downtime cloudnet1004 for a week [[phab:T271058|T271058]]

=== 2021-02-02 ===
* 17:14 dcaro: Changed osd memory limit from 4G to 8G ([[phab:T273649|T273649]])
* 11:00 arturo: icinga-downtime cloudvirt-wdqs1001 for 1 week ([[phab:T273579|T273579]])
* 03:12 andrewbogott: running /usr/local/sbin/wmcs-purge-backups and /usr/local/sbin/wmcs-backup-instances on cloudvirt1024 to see why the backup job paged

=== 2021-01-29 ===
* 15:36 andrewbogott: disabling puppet and some services on eqiad1 cloudcontrol nodes; replacing nova-placement-api with placement-api

=== 2021-01-28 ===
* 19:44 andrewbogott: shutting down cloudcontrol2001-dev because it's in a partially upgraded state; will revive when it's time for Train

=== 2021-01-27 ===
* 00:50 bstorm: icinga-downtime cloudnet1004 for a week [[phab:T271058|T271058]]

=== 2021-01-22 ===
* 16:44 andrewbogott: upgrading designate on cloudvirt1003/1004 to OpenStack 'train'
* 11:29 dcaro: Doing some tests removed cloudcontrol1003 puppet cert, regenerating...

=== 2021-01-21 ===
* 11:35 arturo: merging core router firewall changes https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657439 ([[phab:T209082|T209082]])
* 11:30 arturo: merging core router firewall changes https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657358 ([[phab:T272486|T272486]], [[phab:T209082|T209082]])

=== 2021-01-20 ===
* 10:49 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657302 ([[phab:T209082|T209082]])
* 10:05 dcaro: Everything looks ok, created a new vm with a volume in ceph without issues, and on warnings/errors on ceph status, closing ([[phab:T272303|T272303]])
* 09:55 dcaro: Eqiad ceph cluster uprgaded, doing sanity checks ([[phab:T272303|T272303]])
* 09:46 dcaro: 75% of the eqiad cluster upgraded... continuing ([[phab:T272303|T272303]])
* 09:37 dcaro: 25% of the eqiad cluster upgraded... continuing ([[phab:T272303|T272303]])
* 09:24 dcaro: Mgr daemons upgraded and running, upgrading osd daemons on servers cloudcephosd1*, this make take a bit longer ([[phab:T272303|T272303]])
* 09:22 dcaro: Mon daemons upgraded and running, upgrading mgr daemons on servers cloudcephmon1* ([[phab:T272303|T272303]])
* 09:16 dcaro: Starting eqiad ceph upgrade, upgrading the mon servers cloudcephmon1* ([[phab:T272303|T272303]])
* 09:01 dcaro: Will start the ceph upgrade in 15 min, no downtime nor performance impact is expected ([[phab:T272303|T272303]])

=== 2021-01-19 ===
* 10:17 arturo: icinga-downtime cloudnet1004 for 1 week ([[phab:T271058|T271058]])

=== 2021-01-18 ===
* 16:00 dcaro: Codfw1 ceph cluster uprgaded, will wait until tomorrow to see if there's any instability, but everything looks fine ([[phab:T272303|T272303]])
* 15:38 dcaro: Upgraded mgr sevices on codfw ceph cluster, starting with osd ones ([[phab:T272303|T272303]])
* 15:35 dcaro: Upgraded mon sevices on codfw ceph cluster, starting with mgr ones ([[phab:T272303|T272303]])
* 15:21 dcaro: Starting upgrade of ceph mon nodes on codfw ([[phab:T272303|T272303]])
* 15:06 dcaro: re-enabling puppet on cloudcephosd2* hosts
* 13:53 dcaro: disabling puppet on cloudcephosd2* to resume perf tests
* 10:50 dcaro: re-enabling puppet on cephcloudosd2* (codfw)
* 10:07 dcaro: disabling puppet on cephcloudosd2* (codfw) to do some performance tests
* 09:00 dcaro: Enabling custom application 'cinder' on pool codfw1dev-cinder to get rid of health warnings

=== 2021-01-17 ===
* 16:53 arturo: icinga downtime labstore1004 /srv/tools space check for 3 days ([[phab:T272247|T272247]])

=== 2021-01-15 ===
* 13:41 arturo: icinga downtime labstore1004 maintain-dbuser alert until 2021-01-19 ([[phab:T272125|T272125]])
* 09:47 arturo: labstore1004 maintain-dbusers affected by [[phab:T272127|T272127]] and [[phab:T272125|T272125]]
* 09:22 arturo: restart maintain-dbusers.service in labstore1004
* 08:19 dcaro: Merging the patch to disable write caches on ceph osds ([[phab:T271527|T271527]])

=== 2021-01-13 ===
* 17:03 arturo: remove cloudvirt1013 cloudvirt1032 cloudvirt1037 to the 'toobusy' host aggregate to prevent further CPU oversubscribing
* 12:40 arturo: try increasing systemd watchdog timeout for conntrackd in cloudnet1004 ([[phab:T268335|T268335]])
* 11:45 dcaro: https://gerrit.wikimedia.org/r/c/operations/puppet/+/654419 merged and deployed (and tested) ([[phab:T268877|T268877]])
* 11:40 dcaro: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/654419 that might affect the encapi service (puppet on cloud environment), no downtime expected though ([[phab:T268877|T268877]])
* 10:56 arturo: trying to cleanup dpkg package mess in cloudnet2002-dev
* 10:02 arturo: prevent floating IP allocation from neutron transport subnet: root@cloudcontrol1005:~# neutron subnet-update --allocation-pool start=185.15.56.244,end=185.15.56.244 cloud-instances-transport1-b-eqiad1 ([[phab:T271867|T271867]])

=== 2021-01-12 ===
* 10:33 arturo: reboot cloudnet1004
* 10:32 arturo: update firmware-bnx2x from 20190114-2 to 20200918-1~bpo10+1 on cloudnet1004 ([[phab:T271058|T271058]])

=== 2021-01-11 ===
* 10:22 arturo: doubling size of conntrack table in cloudnet servers https://gerrit.wikimedia.org/r/c/operations/puppet/+/655407 ([[phab:T271058|T271058]])
* 10:07 arturo: manually cleanup conntrack table in cloudnet1004 ([[phab:T271058|T271058]])
* 09:19 dcaro: cleaned up ~1800 snapshots, 109 remaining only, one for each host x image combination (plus some ephemeral ones while doing backups), closing the task ([[phab:T270478|T270478]])
* 08:39 dcaro: cleaning up dangling snapshots now that we have the new suffixed ones ([[phab:T270478|T270478]])

=== 2021-01-10 ===
* 16:02 andrewbogott: restarting rabbitmq-server on all eqiad1 cloudcontrols
* 15:54 andrewbogott: restating neutron-metadata-agent on cloudnet1004 due to many syslog complaints

=== 2021-01-08 ===
* 11:25 arturo: rebooting both cloudnet2002-dev/cloudnet2003-dev to make sure interfaces are set up correctl ([[phab:T271517|T271517]])
* 11:22 arturo: connecting cloudnet2002-dev cloudnet2003-dev back to vlan 2120 ([[phab:T271517|T271517]])
* 11:06 arturo: root@cloudcontrol2001-dev:~# openstack router set --external-gateway wan-transport-codfw --fixed-ip subnet=cloud-instances-transport1-b-codfw,ip-address=208.80.153.190 cloudinstances2b-gw ([[phab:T271517|T271517]])
* 11:02 arturo: root@cloudcontrol2001-dev:~# openstack router set --enable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw ([[phab:T271517|T271517]])
* 11:01 arturo: enabling neutron hacks in codfw1dev (cloudnet2002-dev, cloudnet2003-dev) ([[phab:T271517|T271517]])
* 10:55 arturo: aborrero@labtestvirt2003:~ $ sudo ifdown eno2.2107 ([[phab:T271517|T271517]])
* 10:55 arturo: aborrero@labtestvirt2003:~ $ sudo ifdown eno2.2120 ([[phab:T271517|T271517]])
* 10:53 arturo: root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 208.80.153.185 --ip-version 4 --network wan-transport-codfw --no-dhcp --subnet-range 208.80.153.184/29 cloud-instances-transport1-b-codfw ([[phab:T271517|T271517]])
* 10:40 dcaro: Finished tests, brining osd online (od.48) for eqiad ceph cluster ([[phab:T271417|T271417]])
* 09:59 dcaro: Started performance tests on sdc (od.48) for eqiad ceph cluster ([[phab:T271417|T271417]])
* 09:41 dcaro: Taking osd.48 from eqiad ceph cluster out to do performance tests ([[phab:T271417|T271417]])

=== 2021-01-07 ===
* 15:19 dcaro: Finished speed tests on cloudcephosd2001-dev, reprovisioning the osd.0 sdc ([[phab:T271417|T271417]])
* 14:39 dcaro: Starting speed tests on cloudcephosd2001-dev sdc ([[phab:T271417|T271417]])
* 12:54 dcaro: Taking osd.0 down on codfw ceph cluster to try the disk performance testing process ([[phab:T271417|T271417]])
* 11:35 arturo: merging dmz_cidr change ([[phab:T209082|T209082]], [[phab:T267779|T267779]])

=== 2021-01-05 ===
* 10:40 dcaro: removing dumps-[1..*] backups from cloudvirt1024 as they are not needed ([[phab:T271094|T271094]])

=== 2021-01-03 ===
* 07:06 dcaro: Got a network hiccup on cloudnet1004, keeping track here [[phab:T271058|T271058]]

=== 2020-12-28 ===
* 12:32 arturo: stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 ([[phab:T260692|T260692]])
* 12:32 arturo: stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 ([[phab:T260682|T260682]])
* 12:23 arturo: icinga downtime cloudvirt1026 disk space check until january 5 ([[phab:T260692|T260692]])
* 06:15 andrewbogott: restarting designate-central on cloudservices1003/1004.  I'm pretty sure they're distressed because of DB lag but it's worth a try

=== 2020-12-23 ===
* 15:38 andrewbogott: restarting rabbitmq on cloudcontrol1004; suspected leaks
* 15:33 andrewbogott: restarting each cloudcontrol galera node in turn to see if that quiets down the syncing warnings
* 12:08 arturo: move memory out of the swap in cloudcontrol1004 by disabling/enabling it (1Gb swap was being used)

=== 2020-12-22 ===
* 15:30 dcaro: cleaning up 6778 dangling snapshots for glance images in eqiad ([[phab:T270478|T270478]])
* 13:51 dcaro: merged patch to move wikidumpparse backups to cloudvirt1025 to free space on cloudvirt1026

=== 2020-12-19 ===
* 16:18 dcaro: gzipped a bunch of logs on cloudvirt1004 due to / being out of space
* 00:14 bstorm: truncated /var/log/debug.1 on cloudcontrol1003 which appears to be the exact same content as the user.log files anyway
* 00:10 bstorm: truncated /var/log/daemon.log.1 and the haproxy log
* 00:02 bstorm: truncated /var/log/messages.1 on cloudcontrol1003

=== 2020-12-18 ===
* 23:53 bstorm: truncated haproxy.log.1 on cloudcontrol1003
* 20:46 andrewbogott: setting  pg and pgp number to 4096 for eqiad1-compute as joachim thinks 8192 might be too much [[phab:T270305|T270305]]
* 17:09 dcaro: finished cleaning up the dangling snapshots from cloudvirt1026 ([[phab:T270478|T270478]])
* 17:08 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1026) ([[phab:T270478|T270478]])
* 17:06 dcaro: finished cleaning up the dangling snapshots from cloudvirt1025 ([[phab:T270478|T270478]])
* 17:05 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1025) ([[phab:T270478|T270478]])
* 17:00 dcaro: finished cleaning up the dangling snapshots from cloudvirt1021 ([[phab:T270478|T270478]])
* 16:58 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1021) ([[phab:T270478|T270478]])
* 16:56 dcaro: finished cleaning up the dangling snapshots from cloudvirt1022 ([[phab:T270478|T270478]])
* 16:55 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1022) ([[phab:T270478|T270478]])
* 16:54 dcaro: finished cleaning up the dangling snapshots from cloudvirt1023 ([[phab:T270478|T270478]])
* 16:51 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1023) ([[phab:T270478|T270478]])
* 16:47 dcaro: finished cleaning up the dangling snapshots from cloudvirt1024, freed ~12% of the capacity ([[phab:T270478|T270478]])
* 16:21 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1024) ([[phab:T270478|T270478]])
* 16:13 andrewbogott: setting  autoscale to 'off' for both ceph pools (eqiad1-compute and eqiad1-glance-images) because we like how things are set and the autoscaler does not
* 10:33 dcaro: purging rbd snapshots for image fc6fb78b-4515-4dcc-8254-{{Gerrit|591b9fe01762}} ([[phab:T270478|T270478]])

=== 2020-12-17 ===
* 22:17 andrewbogott: correction to above, set the pg and pgp to 1024 for eqiad1-glance-images
* 22:16 andrewbogott: setting  pgp number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) [[phab:T270305|T270305]] (same as pg)
* 22:14 andrewbogott: setting  pg number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) [[phab:T270305|T270305]]
* 22:10 andrewbogott: setting  autoscale to 'warn' for both ceph pools (eqiad1-compute and eqiad1-glance-images)

=== 2020-12-16 ===
* 09:31 dcaro: removing invalid backups from cloudvirt1024 (196 in total) ([[phab:T269419|T269419]])

=== 2020-12-14 ===
* 17:42 dcaro: The removal freed ~12GB (still 100% usage :S) ([[phab:T269419|T269419]])
* 17:36 dcaro: removing invalid backups that have a valid copy ([[phab:T269419|T269419]])
* 15:43 dcaro: Merging the tagging for vm backups ([[phab:T267195|T267195]])
* 09:45 arturo: icinga downtime cloudvirt1024 for 6 days ([[phab:T269419|T269419]])

=== 2020-12-13 ===
* 09:11 _dcaro: running backup purge script on cloudvirt1024 ([[phab:T269419|T269419]])

=== 2020-12-10 ===
* 23:36 bstorm: cleaned up the logs for haproxy on cloudcontrol1003 by deleting all the gzipped ones and truncating the .1 file
* 11:56 dcaro: Freed some space on cloudvirt1024 by running the purge script ([[phab:T269419|T269419]])
* 09:17 dcaro: removing leaked dns record discordwiki.eqiad.wmflabs (clinic duty)

=== 2020-12-08 ===
* 18:01 dcaro: Host cloudvirt1030 up and running ([[phab:T216195|T216195]])
* 15:59 dcaro: Re-imaging host cloudvirt1030 ([[phab:T216195|T216195]])
* 14:18 dcaro: Host online cloudvirt1029 ([[phab:T216195|T216195]])
* 14:13 dcaro: Host re-imaged, doing tests cloudvirt1029 ([[phab:T216195|T216195]])
* 12:14 dcaro: Re-imaging cloudvirt1029 ([[phab:T216195|T216195]])

=== 2020-12-07 ===
* 18:33 andrewbogott: putting cloudvirt1023 back into service [[phab:T269467|T269467]]
* 15:55 andrewbogott: reimaging cloudvirt1028 for [[phab:T216195|T216195]]
* 14:49 dcaro: Re-imaging cloudvirt1027 ([[phab:T216195|T216195]])

=== 2020-12-05 ===
* 00:35 andrewbogott: moving cloudvirt1023 back into maintenance because [[phab:T269467|T269467]] continues to puzzle

=== 2020-12-04 ===
* 22:33 andrewbogott: moving cloudvirt1023 back into the ceph aggregate; it doesn't need upgrades after all  [[phab:T269467|T269467]]
* 22:24 andrewbogott: moving cloudvirt1023 out of the ceph aggregate and into maintenance for [[phab:T269467|T269467]]
* 21:06 andrewbogott: putting cloudvirt1025 and 1026 back into service because I'm pretty sure they're fixed.  [[phab:T269313|T269313]]
* 12:12 arturo: manually running `wmcs-purge-backups` again on cloudvirt1024 ([[phab:T269419|T269419]])
* 11:25 arturo: icinga downtime cloudvirt1024 for 6 days, to avoid paging noises ([[phab:T269419|T269419]])
* 11:25 arturo: last log line referencing cloudvirt1024 is a mistake ([[phab:T269313|T269313]])
* 11:24 arturo: icinga downtime cloudvirt1024 for 6 days, to avoid paging noises ([[phab:T269313|T269313]])
* 10:28 arturo: manually running `wmcs-purge-backups` on cloudvirt1024 ([[phab:T269419|T269419]])
* 10:23 arturo: setting expiration to 2020-12-03 to the oldest backy snapshot of every VM in cloudvirt1024 ([[phab:T269419|T269419]])
* 09:54 arturo: icinga downtime cloudvirt1025 for 6 days ([[phab:T269313|T269313]])

=== 2020-12-03 ===
* 23:21 andrewbogott: removing all osds on cloudcephosd1004 for rebuild, [[phab:T268746|T268746]]
* 21:45 andrewbogott: removing all osds on cloudcephosd1005 for rebuild, [[phab:T268746|T268746]]
* 19:51 andrewbogott: removing all osds on cloudcephosd1006 for rebuild, [[phab:T268746|T268746]]
* 17:01 arturo: icinga downtime cloudvirt1025 for 48h to debug network issue [[phab:T269313|T269313]]
* 16:56 arturo: rebooting cloudvirt1025 to debug network issue [[phab:T269313|T269313]]
* 16:38 dcaro: Rimaging cloudvirt1026 ([[phab:T216195|T216195]])
* 13:24 andrewbogott: removing all osds on cloudcephosd1008 for rebuild, [[phab:T268746|T268746]]
* 02:55 andrewbogott: removing all osds on cloudcephosd1009 for rebuild, [[phab:T268746|T268746]]

=== 2020-12-02 ===
* 20:04 andrewbogott: removing all osds on cloudcephosd1010 for rebuild, [[phab:T268746|T268746]]
* 17:25 arturo: [15:51] failovering neutron virtual router in eqiad1 ([[phab:T268335|T268335]])
* 15:36 arturo: conntrackd is now up and running in cloudnet1003/1004 nodes ([[phab:T268335|T268335]])
* 15:33 arturo: [codfw1dev] conntrackd is now up and running in cloudnet200x-dev nodes ([[phab:T268335|T268335]])
* 15:08 andrewbogott: removing all osds on cloudcephosd1012 for rebuild, [[phab:T268746|T268746]]
* 12:41 arturo: disable puppet in all cloudnet servers to merge conntrackd change [[phab:T268335|T268335]]
* 11:12 dcaro: Reset the properties for the flavor g2.cores8.ram16.disk1120 to correct quotes ([[phab:T269172|T269172]])
* 09:57 arturo: moved cloudvirts 1030, 1029, 1028, 1027, 1026, 1025 away from the 'standard' host aggregate to 'maintenance' ([[phab:T269172|T269172]])

=== 2020-12-01 ===
* 20:06 andrewbogott: removing all osds on cloudcephosd1014 for rebuild, [[phab:T268746|T268746]]
* 12:04 arturo: restarting neutron l3 agents to pick up config change
* 11:48 arturo: merging change to dmz_dir, detail list of private address https://gerrit.wikimedia.org/r/c/operations/puppet/+/641977

=== 2020-11-30 ===
* 18:12 andrewbogott: removing all osds from cloudcephosd1015 in order to investigate [[phab:T268746|T268746]]

=== 2020-11-29 ===
* 17:18 andrewbogott: cleaning up some logfiles in tools-sgecron-01 — drive is full

=== 2020-11-26 ===
* 22:58 andrewbogott: deleting /var/log/haproxy logs older than 7 days in cloudcontrol100x.  We need log rotation here it seems.
* 15:53 dcaro: Created private flavor g2.cores8.ram16.disk1120 for wikidumpparse ([[phab:T268190|T268190]])

=== 2020-11-25 ===
* 19:35 bstorm: repairing ceph pg `instructing pg 6.91 on osd.117 to repair`
* 09:31 _dcaro: The OSD seems to be up and running actually, though there's that misleading log, will leave it see if the cluster comes fully healthy ([[phab:T268722|T268722]])
* 08:54 _dcaro: Unsetting noup/nodown to allow re-shuffling of the pgs that osd.44 had, will try to rebuild it ([[phab:T268722|T268722]])
* 08:45 _dcaro: Tried resetting the class for osd.44 to ssd, no luck, the cluster is in noout/norebalance to avoid data shuffling (opened [[phab:T268722|T268722]])
* 08:45 _dcaro: Tried resetting the class for osd.44 to ssd, no luck, the cluster is in noout/norebalance to avoid data shuffling (opened root@cloudcephosd1005:/var/lib/ceph/osd/ceph-44# ceph osd crush set-device-class ssd osd.44)
* 08:19 _dcaro: Restarting serivce osd.44 resulted on osd.44 being unable to start due to some config inconsistency (can not reset class to hdd)
* 08:16 _dcaro: After enabling auto pg scaling on ceph eqiad cluster, osd.44 (cloudcephosd1005) got stuck, trying to restart the osd service
* 08:16 _dcaro: After enabling auto pg scaling on ceph eqiad cluster, osd.44 (cloudcephosd1005) got stuck, trying to restart

=== 2020-11-22 ===
* 17:40 andrewbogott: apt-get upgrade on cloudservices1003/1004
* 17:32 andrewbogott: upgrading Designate on cloudservices1003/1004 to Stein

=== 2020-11-20 ===
* 12:44 arturo: [codfw1dev] install conntrackd in cloudnet2003-dev/cloudnet2002-dev to research l3 agent HA reliability
* 09:26 arturo: incinga downtime labstore1006 RAID checks for 10 days ([[phab:T268281|T268281]])

=== 2020-11-17 ===
* 19:21 andrewbogott: draining cloudvirt1012 to experiment with libvirt/cpu things

=== 2020-11-15 ===
* 11:21 arturo: icinga downtime cloudbackup2002 for 48h ([[phab:T267865|T267865]])

=== 2020-11-10 ===
* 16:38 arturo: icinga downtime toolschecker for 2h becasue toolsdb maintenance ([[phab:T266587|T266587]])
* 11:24 arturo: [codfw1dev] enable puppet in puppetmaster01.cloudinfra-codfw1dev (disabled for unspecified reasons)

=== 2020-11-09 ===
* 12:42 arturo: restarted neutron l3 agent in cloudnet1003 bc it still had the old default route ([[phab:T265288|T265288]])
* 12:41 arturo: `root@cloudcontrol1005:~# neutron subnet-delete dcbb0f98-5e9d-4a93-8dfc-4e3ec3c44dcc` ([[phab:T265288|T265288]])
* 12:41 arturo: `root@cloudcontrol1005:~# neutron router-gateway-set  --fixed-ip subnet_id=7c6bcc12-212f-44c2-9954-{{Gerrit|5c55002ee371}},ip_address=185.15.56.244 cloudinstances2b-gw wan-transport-eqiad` ([[phab:T265288|T265288]])
* 12:19 arturo: subnet 185.1.5.56.240/29 has id 7c6bcc12-212f-44c2-9954-{{Gerrit|5c55002ee371}} in neutron ([[phab:T265288|T265288]])
* 12:19 arturo: `root@cloudcontrol1005:~# neutron subnet-create --gateway 185.15.56.241 --name cloud-instances-transport1-b-eqiad1 --ip-version 4 --disable-dhcp wan-transport-eqiad 185.15.56.240/29` ([[phab:T265288|T265288]])
* 12:15 arturo: icinga-downtime toolschecker for 2h ([[phab:T265288|T265288]])

=== 2020-11-02 ===
* 13:36 arturo: (typo: dcaro)
* 13:35 arturo: added dcar as projectadmin &amp; user ([[phab:T266068|T266068]])

=== 2020-10-29 ===
* 16:57 bstorm: silenced deployment-prep project alerts for 60 days since the downtime expired
* 08:12 arturo: force-powercycling cloudcephosd1006

=== 2020-10-25 ===
* 16:20 andrewbogott: adding cloudvirt1038 to the 'ceph' aggregate and removing from the 'spare' aggregate.  We need this space while waiting on network upgrades for empty cloudvirts ([[phab:T216195|T216195]])

=== 2020-10-23 ===
* 11:30 arturo: [codfw1dev] openstack --os-project-id cloudinfra-codfw1dev recordset create --type PTR --record nat.cloudgw.codfw1dev.wikimediacloud.org. --description &quot;created by hand&quot; 0-29.57.15.185.in-addr.arpa. 1.0-29.57.15.185.in-addr.arpa. ([[phab:T261724|T261724]])
* 10:09 arturo: [codf1dev] doing DNS changes for the cloudgw PoC, including designate and https://gerrit.wikimedia.org/r/c/operations/dns/+/635965 ([[phab:T261724|T261724]])

=== 2020-10-22 ===
* 10:46 arturo: [codfw1dev] rebooting cloudinfra-internal-puppetmaster-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud to try fixing some DNS weirdness
* 09:43 arturo: enabling puppet in cloucontrol1003 (message said &quot;please re-enable after 2020-10-22 06:00UTC&quot;)

=== 2020-10-21 ===
* 14:36 andrewbogott: running apt-get update &amp;&amp; apt-get install -y facter on all cloud-vps instances
* 10:31 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])
* 08:56 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])

=== 2020-10-20 ===
* 15:47 arturo: changing DNS recursor ACLs (https://gerrit.wikimedia.org/r/c/operations/puppet/+/635314) this can be reverted any time if it causes problems ([[phab:T261724|T261724]])
* 14:49 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])

=== 2020-10-19 ===
* 01:41 andrewbogott: deleting all Precise base images
* 01:36 andrewbogott: deleting all unused Jessie base images

=== 2020-10-18 ===
* 23:26 andrewbogott: deleting all Trusty base images
* 21:50 andrewbogott: migrating all currently used ceph images to rbd

=== 2020-10-16 ===
* 09:29 arturo: [codfw1dev] still some DNS weirdness, investigating
* 09:25 arturo: [codfw1dev] hard-rebooting bastion-codfw1dev-02, seems in bad shape, doesn't even wake up in the virsh console
* 09:18 arturo: [codfw1dev] live-hacked cloudservices2002-dev /etc/powerdns/recursor.conf file to include cloud-codfw1dev-floating CIDR (185.15.57.0/29) while https://gerrit.wikimedia.org/r/c/operations/puppet/+/634050 is in review, so VMs with a floating IP can query the DNS recursor ([[phab:T261724|T261724]])
* 09:01 arturo: [codfw1dev] basic network connectivity seems stable after cleaning up everything related to address scopes ([[phab:T261724|T261724]])

=== 2020-10-15 ===
* 15:17 arturo: [codfw1dev] try cleaning up anything related to address scopes in the neutron database ([[phab:T261724|T261724]])
* 13:56 arturo: [codfw1dev] drop neutron l3 agent hacks in cloudnet2002/2003-dev ([[phab:T261724|T261724]])

=== 2020-10-13 ===
* 17:54 andrewbogott: rebuilding cloudvirt1021 for backy support
* 15:22 andrewbogott: draining cloudvirt1021 so I can rebuild it with backy support
* 14:19 andrewbogott: rebuilding cloudvirt1022 with backy support
* 14:03 andrewbogott: draining cloudvirt1022 so I can rebuild it with backy support
* 11:19 arturo: [codfw1dev] rebooting labtestvirt2003

=== 2020-10-09 ===
* 10:15 arturo: [codfwd1ev] root@cloudcontrol2001-dev:~# openstack router set --disable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw ([[phab:T261724|T261724]])
* 09:22 arturo: [codfwd1dev] rebooting cloudnet boxes for bridge and vlan changes ([[phab:T261724|T261724]])
* 09:12 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet delete 31214392-9ca5-4256-bff5-{{Gerrit|1e19a35661de}} (cloud-instances-transport1-b-codfw - 208.80.153.184/29) ([[phab:T261724|T261724]])
* 09:10 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack router set --external-gateway wan-transport-codfw --fixed-ip subnet=cloud-gw-transport-codfw,ip-address=185.15.57.10 cloudinstances2b-gw ([[phab:T261724|T261724]])
* 08:49 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.9 --no-dhcp --subnet-range 185.15.57.8/30 cloud-gw-transport-codfw ([[phab:T261724|T261724]])
* 08:47 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet delete a5ab5362-4ffb-4059-9ff7-{{Gerrit|391e22dcf3bc}} ([[phab:T261724|T261724]])

=== 2020-10-08 ===
* 16:17 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.8 --no-dhcp --subnet-range 185.15.57.8/31 cloud-gw-transport-codfw` (with a hack -- see task) ([[phab:T263622|T263622]])
* 16:03 arturo: [codfw1dev] briefly live-hacked python3-neutron source code in all 3 cloudcontrol2xxx-dev servers to workaround /31 network definition issue ([[phab:T263622|T263622]])
* 10:28 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) [[phab:T261724|T261724]]

=== 2020-10-06 ===
* 21:30 andrewbogott: moved cloudvirt1013 out of the 'ceph' aggregate and into the 'maintenance' aggregate for [[phab:T243414|T243414]]
* 21:29 andrewbogott: draining cloudvirt1013 for upgrade to 10G networking
* 14:45 arturo: icinga downtime every cloud* lab* host for 60 minutes for keystone maintenance

=== 2020-10-05 ===
* 17:40 bd808: `service uwsgi-labspuppetbackend restart` on cloud-puppetmaster-03 ([[phab:T264649|T264649]])

=== 2020-10-02 ===
* 11:05 arturo: [codfw1dev] restarting rabbitmq-server in all 3 control nodes, the l3 agent was misbehaving
* 09:16 arturo: [codfw1dev] trying the labtestvirt2003 (cloudgw) reimage again ([[phab:T261724|T261724]])

=== 2020-10-01 ===
* 16:06 arturo: rebooting cloudvirt1024 to validate changes to /etc/network/interfaces file
* 15:36 arturo: [codfw1dev] reimaging labtestvirt2003

=== 2020-09-30 ===
* 16:47 andrewbogott: rebooting cloudvir1032, 1033, 1034 for [[phab:T262979|T262979]]
* 13:28 arturo: enable puppet, reboot and pool back cloudvirt1031
* 13:27 arturo: extend icinga downtimes for another 120 mins
* 13:15 arturo: `aborrero@cloudcontrol1003:~$ sudo nova-manage placement sync_aggregates` after reading a hint in nova-api.log
* 13:02 arturo: rebooting cloudvirt1016 and moving it to the ceph host aggregate
* 12:55 arturo: rebooting cloudvirt1014 and moving it to the ceph host aggregate
* 12:51 arturo: rebooting cloudvirt1013 and moving it to the ceph host aggregate
* 12:39 arturo: root@cloudcontrol1005:~# openstack aggregate add host maintenance cloudvirt1031
* 12:36 arturo: rebooted cloudnet1003 (active) a couple of minutes ago
* 12:36 arturo: move cloudvirt1012 and cloudvirt1039 to the ceph aggregate
* 11:49 arturo: rebooting cloudvirt1039
* 11:46 arturo: rebooting cloudvirt1012
* 11:40 arturo: rebooting cloudnet1004 (standby) to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167 ([[phab:T262979|T262979]])
* 11:38 arturo: [codfw1dev] rebooting cloudnet2002-dev to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167
* 11:36 arturo: [codfw1dev] rebooting cloudnet2003-dev to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167
* 11:33 arturo: disabling puppet and downtiming every virt/net server in the fleet in preparation for merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167 ([[phab:T262979|T262979]])
* 09:32 arturo: rebooting cloudvirt1012 to investigate linuxbridge agent issues

=== 2020-09-29 ===
* 15:40 arturo: downgrade linux kernel from linux-image-4.19.0-11-amd64 to linux-image-4.19.0-10-amd64 on cloudvirt1012
* 14:47 arturo: rebooting cloudvirt1012, chasing config weirdness in the linuxbridge agent
* 14:05 andrewbogott: reimaging 1014 over and over in an attempt to get partman right
* 13:51 arturo: rebooting cloudvirt1012

=== 2020-09-28 ===
* 14:55 arturo: [jbond42] upgraded facter to v3 across the VM fleet
* 13:54 andrewbogott: moving cloudvirt1035 from aggregate 'spare' to 'ceph'.  We're going to need all the capacity we can get while converting older cloudvirts to ceph

=== 2020-09-24 ===
* 15:47 arturo: stopping/restarting rabbitmq-server in all cloudcontrol servers
* 15:45 arturo: restarting rabbitmq-server in cloudcontrol103
* 15:15 arturo: restarting floating_ip_ptr_records_updater.service in all 3 cloudcontrol servers to reset state after a DNS failure

=== 2020-09-18 ===
* 10:16 arturo: cloudvirt1039 libvirtd service issues were fixed with a reboot
* 09:56 arturo: rebooting cloudvirt1039 (spare) to try to fix some weird libvirtd failure
* 09:50 arturo: enabling puppet in cloudvirts and effectively merging patches from [[phab:T262979|T262979]]
* 08:59 arturo: disable puppet in all buster cloudvirts (cloudvirt[1024,1031-1039].eqiad.wmnet) to merge a patch for [[phab:T263205|T263205]] and [[phab:T262979|T262979]]
* 08:50 arturo: installing iptables from buster-bpo in cloudvirt1036 ([[phab:T263205|T263205]] and [[phab:T262979|T262979]])

=== 2020-09-15 ===
* 20:32 andrewbogott: rebooting cloudvirt1038 to see if it resolves [[phab:T262979|T262979]]
* 13:58 andrewbogott: draining cloudvirt1002 with wmcs-ceph-migrate

=== 2020-09-14 ===
* 14:21 andrewbogott: draining cloudvirt1001, migrating all VMs with wmcs-ceph-migrate
* 10:41 arturo: [codfw1dev] trying to get the bonding working for labtestvirt2003 ([[phab:T261724|T261724]])
* 09:47 arturo: installed qemu security update in eqiad1 cloudvirts ([[phab:T262386|T262386]])
* 09:43 arturo: [codfw1dev] installed qemu security update in codfw1dev cloudvirts ([[phab:T262386|T262386]])

=== 2020-09-09 ===
* 18:13 andrewbogott: restarting ceph-mon@cloudcephmon1003 in hopes that the slow ops reported are phantoms
* 18:01 andrewbogott: restarting ceph-mgr@cloudcephmon1003 in hopes that the slow ops reported are phantoms (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/EOWNO3MDYRUZKAK6RMQBQ5WBPQNLHOPV/)
* 17:40 andrewbogott: giving ceph pg autoscale another chance: ceph osd pool set eqiad1-compute pg_autoscale_mode on
* 00:05 bd808: Running wmcs-novastats-dnsleaks ([[phab:T262359|T262359]])

=== 2020-09-08 ===
* 21:48 bd808: Renamed FQDN prefixes to wikimedia.cloud scheme in cloudinfra-db01's labspuppet db ([[phab:T260614|T260614]])
* 14:29 andrewbogott: restarting nova-compute on all cloudvirts (everyone is upset from the reset switch failure)
* 14:18 arturo: restarting nova-fullstack service in cloudcontrol1003
* 14:17 andrewbogott: stopping apache2 on labweb1001 to make sure the Horizon outage is total

=== 2020-09-03 ===
* 09:31 arturo: icinga downtime cloud* servers for 30 mins ([[phab:T261866|T261866]])

=== 2020-09-02 ===
* 08:46 arturo: [codfw1dev] reimaging spare server labtestvirt2003 as debian buster ([[phab:T261724|T261724]])

=== 2020-09-01 ===
* 18:18 andrewbogott: adding drives on cloudcephosd100[3-5] to ceph osd pool
* 13:40 andrewbogott: adding drives on cloudcephosd101[0-2] to ceph osd pool
* 13:35 andrewbogott: adding drives on cloudcephosd100[1-3] to ceph osd pool
* 11:27 arturo: [codfw1dev] rebooting again cloudnet2002-dev after some network tests, to reset initial state ([[phab:T261724|T261724]])
* 11:09 arturo: [codfw1dev] rebooting cloudnet2002-dev after some network tests, to reset initial state ([[phab:T261724|T261724]])
* 10:49 arturo: disable puppet in cloudnet servers to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/623569/

=== 2020-08-31 ===
* 23:26 bd808: Removed stale lockfile at cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs:/var/lib/puppet/volatile/GeoIP/.geoipupdate.lock
* 11:20 arturo: [codfw1dev] livehacking https://gerrit.wikimedia.org/r/c/operations/puppet/+/615161 in the puppetmasters for tests before merging

=== 2020-08-28 ===
* 20:12 bd808: Running `wmcs-novastats-dnsleaks --delete` from cloudcontrol1003

=== 2020-08-26 ===
* 17:12 bstorm: Running 'ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf &quot;%k KB %p\n&quot; &gt; tools_large_files_20200826.txt' on labstore1004 [[phab:T261336|T261336]]

=== 2020-08-21 ===
* 21:34 andrewbogott: restarting nova-compute on cloudvirt1033; it seems stuck

=== 2020-08-19 ===
* 14:21 andrewbogott: rebooting cloudweb2001-dev, labweb1001, labweb1002 to address mediawiki-induced memleak

=== 2020-08-06 ===
* 21:02 andrewbogott: removing cloudvirt1004/1006 from nova's list of hypervisors; rebuilding them to  use as backup test hosts
* 20:06 bstorm: manually stopped the RAID check on cloudcontrol1003 [[phab:T259760|T259760]]

=== 2020-08-04 ===
* 18:54 bstorm: restarting mariadb on cloudcontrol1004 to setup parallel replication

=== 2020-08-03 ===
* 17:02 bstorm: increased db connection limit to 800 across galera cluster because we were clearly hovering at limit

=== 2020-07-31 ===
* 19:28 bd808: wmcs-novastats-dnsleaks --delete (lots of leaked fullstack-monitoring records to clean up)

=== 2020-07-27 ===
* 22:17 andrewbogott: ceph osd pool set compute pg_num 2048
* 22:14 andrewbogott: ceph osd pool set compute pg_autoscale_mode off

=== 2020-07-24 ===
* 19:15 andrewbogott: ceph mgr module enable pg_autoscaler
* 19:15 andrewbogott: ceph osd pool set compute pg_autoscale_mode on

=== 2020-07-22 ===
* 08:55 jbond42: [codfw1dev] upgrading hiera to version5
* 08:48 arturo: [codfw1dev] add jbond as user in the bastion-codfw1dev and cloudinfra-codfw1dev projects
* 08:45 arturo: [codfw1dev] enabled account creation in labtestwiki briefly for jbond42 to create an account

=== 2020-07-16 ===
* 10:48 arturo: merging change to neutron dmz_cidr https://gerrit.wikimedia.org/r/c/operations/puppet/+/613123 ([[phab:T257534|T257534]])

=== 2020-07-15 ===
* 23:15 bd808: Removed Merlijn van Deen from toollabs-trusted Gerrit group ([[phab:T255697|T255697]])
* 11:48 arturo: [codfw1dev] created DNS records (A and PTR) for bastion.bastioninfra-codfw1dev.codfw1dev.wmcloud.org &lt;-&gt; 185.15.57.2
* 11:41 arturo: [codfw1dev] add myself as projectadmin to the `bastioninfra-codfw1dev` project
* 11:39 arturo: [codfw1dev] created DNS zone `bastioninfra-codfw1dev.codfw1dev.wmcloud.org.` in the cloudinfra-codfw1dev project and then transfer ownership to the bastioninfra-codfw1dev project

=== 2020-07-14 ===
* 15:19 arturo: briefly set root@cloudnet1003:~ # sysctl net.ipv4.conf.all.accept_local=1 (in neutron qrouter netns) ([[phab:T257534|T257534]])
* 10:43 arturo: icinga downtime cloudnet* hosts for 30 mins to introduce new check https://gerrit.wikimedia.org/r/c/operations/puppet/+/612390 ([[phab:T257552|T257552]])
* 04:01 andrewbogott: added a wildcard *.wmflabs.org domain pointing at the domain proxy in project-proxy
* 04:00 andrewbogott: shortened the ttl on .wmflabs.org. to 300

=== 2020-07-13 ===
* 16:17 arturo: icinga downtime cloudcontrol[1003-1005].wikimedia.org for 1h for galera database movements

=== 2020-07-12 ===
* 17:39 andrewbogott: switched eqiad1 keystone from m5 to cloudcontrol galera

=== 2020-07-10 ===
* 20:26 andrewbogott: disabling nova api to move database to galera

=== 2020-07-09 ===
* 11:23 arturo: [codfw1dev] rebooting cloudnet2003-dev again for testing sysct/puppet behavior ([[phab:T257552|T257552]])
* 11:11 arturo: [codfw1dev] rebooting cloudnet2003-dev for testing sysct/puppet behavior ([[phab:T257552|T257552]])
* 09:16 arturo: manually increasing sysctl value of net.nf_conntrack_max in cloudnet servers ([[phab:T257552|T257552]])

=== 2020-07-06 ===
* 15:16 arturo: installing 'aptitude' in all cloudvirts

=== 2020-07-03 ===
* 12:51 arturo: [codfw1dev] galera cluster should be up and running, openstack happy ([[phab:T256283|T256283]])
* 11:44 arturo: [codfw1dev] restoring glance database backup from bacula into cloudcontrol2001-dev ([[phab:T256283|T256283]])
* 11:39 arturo: [codfw1dev] stopped mysql database in the galera cluster [[phab:T256283|T256283]]
* 11:36 arturo: [codfw1dev] dropped glance database in the galera cluster [[phab:T256283|T256283]]

=== 2020-07-02 ===
* 15:41 arturo: `sudo wmcs-openstack --os-compute-api-version 2.55 flavor create --private --vcpus 8 --disk 300 --ram 16384 --property aggregate_instance_extra_specs:ceph=true --description &quot;for packaging envoy&quot; bigdisk-ceph` ([[phab:T256983|T256983]])

=== 2020-06-29 ===
* 14:24 arturo: starting rabbitmq-server in all 3 cloudcontrol servers
* 14:23 arturo: stopping rabbitmq-server in all 3 cloudcontrol servers

=== 2020-06-18 ===
* 20:38 andrewbogott: rebooting cloudservices2003-dev due to a mysterious 'host down' alert on a secondary ip

=== 2020-06-16 ===
* 15:38 arturo: created by hand neutron port 9c0a9a13-e409-49de-9ba3-{{Gerrit|bc8ec4801dbf}} `paws-haproxy-vip` ([[phab:T295217|T295217]])

=== 2020-06-12 ===
* 13:23 arturo: DNS zone `paws.wmcloud.org` transferred to the PAWS project ([[phab:T195217|T195217]])
* 13:20 arturo: created DNS zone `paws.wmcloud.org` ([[phab:T195217|T195217]])

=== 2020-06-11 ===
* 19:19 bstorm_: proceeding with failback to labstore1004 now that DRBD devices are consistent [[phab:T224582|T224582]]
* 17:22 bstorm_: delaying failback labstore1004 for drive syncs [[phab:T224582|T224582]]
* 17:17 bstorm_: failing NFS back to labstore1004 to complete the upgrade process [[phab:T224582|T224582]]
* 16:15 bstorm_: failing over NFS for labstore1004 to labstore1005 [[phab:T224582|T224582]]

=== 2020-06-10 ===
* 16:09 andrewbogott: deleting all old cloud-ns0.wikimedia.org and cloud-ns1.wikimedia.org ns records in designate database [[phab:T254496|T254496]]

=== 2020-06-09 ===
* 15:25 arturo: icinga downtime everything cloud* lab* for 2h more ([[phab:T253780|T253780]])
* 14:09 andrewbogott: stopping puppet, all designate services and all pdns services on cloudservices1004 for [[phab:T253780|T253780]]
* 14:01 arturo: icinga downtime everything cloud* lab* for 2h ([[phab:T253780|T253780]])

=== 2020-06-05 ===
* 15:08 andrewbogott: trying to re-enable puppet without losing cumin contact, as per https://phabricator.wikimedia.org/T254589

=== 2020-06-04 ===
* 14:24 andrewbogott: disabling puppet on all instances for /labs/private recovery
* 14:23 arturo: disabling puppet on all instances for /labs/private recovery

=== 2020-05-28 ===
* 23:02 bd808: `/usr/local/sbin/maintain-dbusers --debug harvest-replicas` ([[phab:T253930|T253930]])
* 13:36 andrewbogott: rebuilding cloudservices2002-dev with Buster
* 00:33 andrewbogott: shutting down cloudservices2002-dev to see if we can live without it.  This is in anticipation or rebuilding it entirely for [[phab:T253780|T253780]]

=== 2020-05-27 ===
* 23:29 andrewbogott: disabling the backup job on cloudbackup2001 (just like last week) so the backup doesn't start while Brooke is rebuilding labstore1004 tomorrow.
* 06:03 bd808: `systemctl start mariadb` on clouddb1001 following reboot (take 2)
* 05:58 bd808: `systemctl start mariadb` on clouddb1001 following reboot
* 05:53 bd808: Hard reboot of clouddb1001 via Horizon. Console unresponsive.

=== 2020-05-25 ===
* 16:35 arturo: [codfw1dev] created zone `0-29.57.15.185.in-addr.arpa.` ([[phab:T247972|T247972]])

=== 2020-05-21 ===
* 19:23 andrewbogott: disabling puppet on cloudbackup2001 to prevent the backup job from starting during maintenance
* 19:16 andrewbogott: systemctl disable block_sync-tools-project.service on cloudbackup2001.codfw.wmnet to avoid stepping on current upgrade
* 15:48 andrewbogott: re-imaging cloudnet1003 with Buster

=== 2020-05-19 ===
* 22:59 bd808: `apt-get install mariadb-client` on cloudcontrol1003
* 21:12 bd808: Migrating wcdo.wcdo.eqiad.wmflabs to cloudvirt1023 ([[phab:T251065|T251065]])

=== 2020-05-18 ===
* 21:37 andrewbogott: rebuilding cloudnet2003-dev with Buster

=== 2020-05-15 ===
* 22:10 bd808: Added reedy as projectadmin in cloudinfra project ([[phab:T249774|T249774]])
* 22:05 bd808: Added reedy as projectadmin in admin project ([[phab:T249774|T249774]])
* 18:44 bstorm_: rebooting cloudvirt-wdqs1003 [[phab:T252831|T252831]]
* 15:47 bd808: Manually running wmcs-novastats-dnsleaks from cloudcontrol1003 ([[phab:T252889|T252889]])

=== 2020-05-14 ===
* 23:28 bstorm_: downtimed cloudvirt1004/6 and cloudvirt-wdqs1003 until tomorrow around this time [[phab:T252831|T252831]]
* 22:21 bstorm_: upgrading qemu-system-x86 on cloudvirt1006 to backports version [[phab:T252831|T252831]]
* 22:15 bstorm_: changing /etc/libvirt/qemu.conf and restarting libvirtd on cloudvirt1006 [[phab:T252831|T252831]]
* 21:12 andrewbogott: rebuilding cloudvirt1003-wdqs as part of [[phab:T252831|T252831]]
* 15:47 andrewbogott: moving cloudvirt1004 and cloudvirt1006 to the 'ceph' aggregate for [[phab:T252784|T252784]]
* 15:02 andrewbogott: moving all of cloudvirt100[1-9] into the 'toobusy' host aggregate.  These are slower, have spinning disks, and are due for replacement.

=== 2020-05-12 ===
* 20:33 andrewbogott: moving cloudvirt1023 to the 'standard' pool and out of the 'spare' pool
* 19:10 jeh: disable neutron-openvswitch-agent service on cloudvirt2001-dev.codfw [[phab:T248881|T248881]]
* 19:09 jeh: Shutdown the unused eno2 network interface on cloudvirt2001-dev.codfw to clear up monitoring errors [[phab:T248425|T248425]]
* 18:20 andrewbogott: moving cloudvirt1024 out of the 'maintenance' aggregate and into 'spare'
* 16:45 andrewbogott: restarting neutron-l3-agent on cloudnet1004 so it knows about all three cloudcontrols.  Leaving cloudnet1003 since restarting it there will cause network interruptions
* 14:06 arturo: icinga downtime everything for 2h for Debian Buster migration in some cloud components

=== 2020-05-09 ===
* 16:53 andrewbogott: rebuilding cloudcontrol2001-dev and 2003-dev with buster for [[phab:T252121|T252121]]

=== 2020-05-08 ===
* 19:02 bstorm_: moving tools-k8s-haproxy-2 from cloudvirt1021 to cloudvirt1017 to improve spread

=== 2020-05-05 ===
* 13:58 andrewbogott: rebuilding cloudcontrol2004-dev to test new puppet changes

=== 2020-05-04 ===
* 09:04 arturo: [codfw1dev] manually modify iptables ruleset to only allow SSH from WMF bastions on cloudservices2003-dev and cloudcontrol2004-dev ([[phab:T251604|T251604]])

=== 2020-04-21 ===
* 22:12 andrewbogott: moving cloudvirt1004 out of the 'standard' aggregate and into the 'maintenance' aggregate
* 16:01 jeh: restart cloudceph mon and osd services for openssl upgrades

=== 2020-04-15 ===
* 18:44 jeh: create indexes and views for grwikimedia [[phab:T245912|T245912]]

=== 2020-04-13 ===
* 15:07 jeh: restart memcached on labwebs to increase cache size [[phab:T145703|T145703]]

=== 2020-04-09 ===
* 19:57 andrewbogott: upgrading eqiad1 designate to rocky
* 16:52 andrewbogott: cleaned up a bunch of leaked .eqiad.wmflabs dns records

=== 2020-04-08 ===
* 19:20 andrewbogott: rotated password and api token for pdns servers on cloudservices1003 and cloudservices1004
* 14:54 arturo: `root@cloudcontrol1003:~# cp /etc/inputrc .inputrc` to solve some bash shortcut weirdness

=== 2020-04-07 ===
* 20:57 andrewbogott: service sssd stop; rm -rf /var/lib/sss/db*; service sssd start on tools-sgebastion-08

=== 2020-04-06 ===
* 22:39 andrewbogott: deleting bogus groups cn=b'project-bastion',ou=groups,dc=wikimedia,dc=org and cn=b'project-tools',ou=groups,dc=wikimedia,dc=org from ldap
* 17:42 arturo: [codfw1dev] transferred DNS zone 57.15.185.in-addr.arpa. to the cloudinfra-codfw1dev project ([[phab:T247972|T247972]])
* 17:39 arturo: [codfw1dev] `openstack zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description &quot;floating IPs subnet&quot; 57.15.185.in-addr.arpa.` ([[phab:T247972|T247972]])
* 16:23 arturo: restarting apache2 in cloudcontrol1003/1004 to pick up latest wmfkeystonehooks changes [[phab:T249494|T249494]]

=== 2020-04-02 ===
* 20:59 jeh: codfw1dev clear VM error states and start bastions, puppet master and database

=== 2020-04-01 ===
* 16:27 arturo: [codfw1dev] enable puppet across the fleet clean vxlan changes ([[phab:T248881|T248881]])

=== 2020-03-31 ===
* 12:35 arturo: [codfw1dev] restarting VMs: designaterockytest14, bastion-codfw1dev-0[1,2] ([[phab:T248881|T248881]])
* 12:34 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudvirt2001-dev ([[phab:T248881|T248881]])
* 12:25 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudnet200[2,3]-dev ([[phab:T248881|T248881]])
* 11:45 arturo: [codfw1dev] rebooting cloudvirt2003-dev to pick up latest kernel update. Otherwise modprobe is confused trying to load modules and openvswitch won't start ([[phab:T248881|T248881]])
* 10:40 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudvirt2003-dev ([[phab:T248881|T248881]])
* 10:09 arturo: [codfw1dev] reboot cloudnet2003-dev into linux 4.9 (was using 4.14 from a testing operation in 2020-03-10)

=== 2020-03-30 ===
* 23:42 bstorm_: deleted &quot;Kubernetes Cluster&quot; and &quot;Kubernetes Performance&quot; dashboards [[phab:T246689|T246689]]
* 16:44 arturo: [codfw1dev] installing package neutron-openvswitch-agent in cloudvirt2002-dev ([[phab:T248881|T248881]])
* 16:42 andrewbogott: restarting l3 agents on cloudnets in codfw1dev after applying https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/584188/

=== 2020-03-27 ===
* 21:28 bd808: Created huggle.wmcloud.org Designate zone and allocated it to the huggle project
* 19:51 jeh: start haproxy on cloudcontrol2003-dev.wikimedia.org

=== 2020-03-26 ===
* 15:01 arturo: icinga downtime cloudvirt* cloudcontrol* cloudnet* lab* cloudstore*
* 15:01 andrewbogott: beginning openstack upgrade window for [[phab:T242766|T242766]]
* 12:32 arturo: [codfw1dev] downgraded systemd, libsystemd0, udev and friends to the non-backports versions ([[phab:T247013|T247013]])

=== 2020-03-25 ===
* 19:29 andrewbogott: dumping a bunch of VMs on cloudvirt1015 to see if it still crashes
* 17:56 jeh: add labweb1002 back into the pool - completed horizon testing [[phab:T240852|T240852]]
* 17:09 jeh: depool labweb1002 for horizon testing [[phab:T240852|T240852]]

=== 2020-03-24 ===
* 19:41 jeh: switch cloudvirt1016 from maintenance to standard host aggregate [[phab:T243327|T243327]]
* 15:31 andrewbogott: restarting nova-conductor and nova-api on cloudcontrol1003 and cloudcontrol1004

=== 2020-03-23 ===
* 21:41 jeh: restart neutron-l3-agent on cloudnet100[3,4] to pickup policy.yaml changes
* 13:28 jeh: disable puppet on labweb100[1,2] to enable horizon event traces [[phab:T240852|T240852]]
* 10:26 arturo: restarting apache in both labweb1001/labweb1002 upon reports of returning 500s

=== 2020-03-21 ===
* 14:23 andrewbogott: restarting apache2 on labweb1001 and 1002

=== 2020-03-18 ===
* 19:17 andrewbogott: deleted a bunch of records from the pdns database on cloudservices1003/1004 which had a record name but the content (where an IP address should be) was NULL, e.g. m.wikidata.beta.wmflabs.org.
* 10:55 arturo: [codfw1dev] deleting BGP agent, undoing changes we did for [[phab:T245606|T245606]]

=== 2020-03-14 ===
* 17:40 jeh: restart maintain-dbusers on labstore1004 [[phab:T247654|T247654]]

=== 2020-03-13 ===
* 12:39 arturo: [codfw1dev] reintroduce address scopes for another round of testing [[phab:T244851|T244851]]
* 12:17 arturo: [codfw1dev] enabling puppet in cloudnet200x-dev servers after merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/579259 ([[phab:T247505|T247505]])

=== 2020-03-12 ===
* 22:29 bstorm_: running puppet across all dumps mounts to make sure active links are shifted to labstore1006

=== 2020-03-11 ===
* 18:38 jeh: set icingia downtime until 2020-03-23 on CODFW cloud[control,net,virt] hosts during openstack upgrades
* 12:50 arturo: [codfw1dev] several tests creating/deleting address scopes ([[phab:T244727|T244727]] [[phab:T247135|T247135]] [[phab:T246887|T246887]] [[phab:T245606|T245606]])
* 12:46 arturo: [codfw1dev] disable routing_source_ip in l3 agents for testing proposal detailed at https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Network_refresh#Eliminate_routing_source_ip_address ([[phab:T244727|T244727]])

=== 2020-03-10 ===
* 17:02 arturo: [codfw1dev] deleting address scopes, bad interaction with our custom NAT setup [[phab:T247135|T247135]]
* 13:55 arturo: [codfw1dev] rebooting cloudnet2003-dev into linux kernel 4.14 for testing stuff related to [[phab:T247135|T247135]]

=== 2020-03-09 ===
* 18:09 arturo: enabling puppet in cloudvirt1006, all services have been restored
* 17:59 arturo: deleted the neutron bridge on cloudvirt1006, for testing stuff related to the queens upgrade
* 17:58 arturo: stopped neutron-linuxbridge-agent and nova-compute in cloudvirt1006 for testing stuff related to the queens upgrade

=== 2020-03-06 ===
* 14:54 andrewbogott: draining all instances off of cloudvirt1006 for [[phab:T246908|T246908]]

=== 2020-03-05 ===
* 14:24 arturo: [codfw1dev] we just enabled BGP session between cloudnet2xxx-dev and cr1-codfw ([[phab:T245606|T245606]])
* 13:07 arturo: [codfw1dev] move the extra IP address for BGP in cloudnet200x-dev servers from eno2.2120 to the br-external bridge device ([[phab:T245606|T245606]])
* 13:06 arturo: [codfw1dev] upgrade neutron-dynamic-routing packages in cloudnet200X-dev and cloudcontrol200X-dev servers to 11.0.0-2~bpo9+1 ([[phab:T245606|T245606]])

=== 2020-03-04 ===
* 22:22 andrewbogott: upgrading designate on cloudservices1003/1004 to Queens
* 22:09 andrewbogott: moving cloudvirt1006 into the maintenance aggregate for [[phab:T246908|T246908]]
* 21:37 bd808: Running wmcs-wikireplica-dns to add service names for ngwikimedia.*.db.svc.eqiad.wmflabs ([[phab:T240772|T240772]])
* 21:14 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1009 ([[phab:T246056|T246056]])
* 21:11 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1010 ([[phab:T246056|T246056]])
* 21:08 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1011 ([[phab:T246056|T246056]])
* 21:05 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1002 ([[phab:T246056|T246056]])

=== 2020-03-02 ===
* 16:54 arturo: [codfw1dev] deleted python3-os-ken debian package in cloudnet2003-dev which was installed by hand and had depedency issues

=== 2020-02-29 ===
* 16:32 bstorm_: downtimed the smart alert on cloudvirt1009 until Monday since apparently predictive failures flap [[phab:T244986|T244986]]

=== 2020-02-26 ===
* 22:03 jeh: powering down cloudvirt1014 for hardware maintenance

=== 2020-02-25 ===
* 16:08 andrewbogott: changing neutron's rabbitmq password because oslo is having trouble parsing some of the characters in the password
* 15:26 andrewbogott: updated the cell_mapping record in the nova_api database to add the second rabbitmq server  to the transport_url field
* 15:26 andrewbogott: updated the cell_mapping record in the nova_api database to set the db uri to 'mysql+pymysql' -- this in response to a deprecation notice

=== 2020-02-24 ===
* 12:16 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker cr2-codfw` ([[phab:T245606|T245606]])
* 12:16 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker cr1-codfw` ([[phab:T245606|T245606]])
* 12:09 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.187 --remote-as 65002 cr2-codfw` ([[phab:T245606|T245606]])
* 12:09 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.186 --remote-as 65002 cr1-codfw` ([[phab:T245606|T245606]])
* 12:06 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-delete 17b8c2a3-f0ce-4d50-a265-18ccac703c61` ([[phab:T245606|T245606]])
* 10:59 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker bgppeer` ([[phab:T245606|T245606]])
* 10:56 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.185 --remote-as 65002 bgppeer` ([[phab:T245606|T245606]])

=== 2020-02-21 ===
* 12:48 arturo: [codfw1dev] running `root@cloudcontrol2001-dev:~# neutron bgp-speaker-network-add bgpspeaker wan-transport-codfw` ([[phab:T245606|T245606]])
* 12:46 arturo: [codfw1dev] created bgpspeaker for AS64711 ([[phab:T245606|T245606]])
* 12:42 arturo: [codfw1dev] run `sudo neutron-db-manage upgrade head` to upgrade the db schema for neutron bgp tables
* 11:51 arturo: [codfw1dev] create a neutron subnet pool per each subnet objects we have and manually update DB to inter-associate them ([[phab:T245606|T245606]])
* 11:49 arturo: [codfw1dev] rename neutron address scope `no-nat` to `bgp` ([[phab:T245606|T245606]])
* 11:37 arturo: [codfw1dev] cleanup unused neutron subnet pools from previous address scope tests ([[phab:T244851|T244851]])

=== 2020-02-20 ===
* 19:22 andrewbogott: updating designate pool config for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/572213/
* 15:33 andrewbogott: migrating all VMs on cloudvirt1014 to cloudvirt1022
* 13:35 arturo: [codfw1dev] disable puppet in cloudcontrol servers to hack neutron.conf for tests related to [[phab:T245606|T245606]]
* 13:33 arturo: [codfw1dev] disable puppet in cloudnet servers to hack neutron.conf for tests related to [[phab:T245606|T245606]]

=== 2020-02-18 ===
* 22:19 andrewbogott: transferred the tools.wmcloud.org. to the tools project
* 22:16 andrewbogott: moved wmcloud.org dns domain to the cloud-infra project
* 21:02 andrewbogott: adding .eqiad1.wikimedia.cloud records to all existing eqiad1 VMs, updating all eqiad1 internal pointer records to reference the new eqiad1.wikimedia.cloud fqdns.
* 09:44 arturo: deleted DNS zone wmcloud.org and try re-creating it

=== 2020-02-14 ===
* 10:35 arturo: running `root@cloudcontrol2001-dev:~# designate server-create --name ns1.openstack.codfw1dev.wikimediacloud.org.` ([[phab:T243766|T243766]])
* 10:32 arturo: running `root@cloudcontrol1004:~# designate server-create --name ns1.openstack.eqiad1.wikimediacloud.org.` ([[phab:T243766|T243766]])
* 10:32 arturo: running `root@cloudcontrol1004:~# designate server-create --name ns0.openstack.eqiad1.wikimediacloud.org.` ([[phab:T243766|T243766]])

=== 2020-02-12 ===
* 13:38 arturo: [codfw1dev] add reference to subnetpool to the instance subnet `MariaDB [neutron]&gt; update subnets set subnetpool_id='d129650d-d4be-4fe1-b13e-6edb5565cb4a' where id = '7adfcebe-b3d0-4315-92fe-e8365cc80668';` ([[phab:T244851|T244851]])

=== 2020-02-11 ===
* 13:46 arturo: [codfw1dev] creating some neutron objects to investigate [[phab:T244851|T244851]] (subnets, subnet pools, address scopes, ...)
* 12:40 arturo: [codfw1dev] delete unknown address scope 'wmcs-v4-scope': `root@cloudcontrol2001-dev:~# openstack address scope delete 078cfd71-117b-4aac-9197-6ebbbb7dd3de` ([[phab:T244851|T244851]])
* 12:40 arturo: [codfw1dev] delete unknown subnet pool 'cloudinstancesb-v4-pool0': `root@cloudcontrol2001-dev:~# openstack subnet pool delete d23a9b88-5c3d-4a53-ab88-053233a75365` ([[phab:T244851|T244851]])

=== 2020-02-07 ===
* 18:11 jeh: shutdown cloudvirt1016 for hardware maintenance [[phab:T241882|T241882]]

=== 2020-02-06 ===
* 14:44 jeh: update apt packages on cloudvirt1015 [[phab:T220853|T220853]]
* 14:28 jeh: run hardware tests on cloudvirt1015 [[phab:T220853|T220853]]

=== 2020-01-28 ===
* 17:24 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# designate server-create --name ns0.openstack.codfw1dev.wikimediacloud.org. ([[phab:T243766|T243766]])
* 10:18 arturo: [codfw1dev] created DNS record `bastion-codfw1dev-01.codfw1dev.wmcloud.org A 185.15.57.2` ([[phab:T242976|T242976]], [[phab:T229441|T229441]])
* 10:13 arturo: [codfw1dev] the zone `codfw1dev.wmcloud.org` belongs now to the `cloudinfra-codfw1dev` project ([[phab:T242976|T242976]])
* 10:11 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description &quot;main DNS domain for public addresses&quot; --email &quot;root@wmflabs.org&quot; --type PRIMARY --ttl 3600 codfw1dev.wmcloud.org.` ([[phab:T242976|T242976]] and [[phab:T243766|T243766]])
* 09:53 arturo: restart apache2 in labweb1001/1002 because horizon errors
* 09:47 arturo: created DNS zone wmcloud.org in eqiad1, transfer it to the cloudinfra project ([[phab:T242976|T242976]]) right now only use is to delegate codfw1dev.wmcloud.org subdomain to designate in the other deployment

=== 2020-01-27 ===
* 12:45 arturo: [codfw1dev] manually move the new domain to the `cloudinfra-codfw1dev` project clouddb2001-dev: `[designate]&gt; update zones set tenant_id='cloudinfra-codfw1dev' where id = '4c75410017904858a5839de93c9e8b3d';` [[phab:T243556|T243556]]
* 12:44 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description &quot;main DNS domain for VMs&quot; --email &quot;root@wmflabs.org&quot; --type PRIMARY --ttl 3600 codfw1dev.wikimedia.cloud.` [[phab:T243556|T243556]]

=== 2020-01-24 ===
* 15:10 jeh: remove icinga downtime for cloudvirt1013 [[phab:T241313|T241313]]
* 12:52 arturo: repooling cloudvirt1013 after HW got fixed ([[phab:T241313|T241313]])

=== 2020-01-21 ===
* 17:43 bstorm_: remounting /mnt/nfs/dumps-labstore1007.wikimedia.org/ on all dumps-mounting projects
* 10:24 arturo: running `sudo systemctl restart apache2.service` in both labweb servers to try mitigating [[phab:T240852|T240852]]

=== 2020-01-15 ===
* 16:59 bd808: Changed the config for cloud-announce mailing list so that lsit admins do not get bounce unsubscribe notices

=== 2020-01-14 ===
* 14:03 arturo: icinga downtime all cloudvirts for another 2h for fixing some icinga checks
* 12:04 arturo: icinga downtime toolchecker for 2 hours for openstack upgrades [[phab:T241347|T241347]]
* 12:02 arturo: icinga downtime cloud* labs* hosts for 2 hours for openstack upgrades [[phab:T241347|T241347]]
* 04:26 andrewbogott: upgrading designate on cloudservices1003/1004

=== 2020-01-13 ===
* 13:34 arturo: [¢odfw1dev] prevent neutron from allocating floating IPs from the wrong subnet by doing `neutron subnet-update --allocation-pool start=208.80.153.190,end=208.80.153.190 cloud-instances-transport1-b-codfw` ([[phab:T242594|T242594]])

=== 2020-01-10 ===
* 13:27 arturo: cloudvirt1009: virsh undefine i-000069b6. This is tools-elastic-01 which is running on cloudvirt1008 (so, leaked on cloudvirt1009)

=== 2020-01-09 ===
* 11:12 arturo: running `MariaDB [nova_eqiad1]&gt; update quota_usages set in_use='0' where project_id='etytree';` ([[phab:T242332|T242332]])
* 11:11 arturo: running `MariaDB [nova_eqiad1]&gt; select * from quota_usages where project_id = 'etytree';` ([[phab:T242332|T242332]])
* 10:32 arturo: ran `root@cloudcontrol1004:~# nova-manage project quota_usage_refresh --project etytree`

=== 2020-01-08 ===
* 10:53 arturo: icinga downtime all cloudvirts for 30 minutes to re-create all canary VMs&quot;

=== 2020-01-07 ===
* 11:12 arturo: icinga-downtime everything cloud* for 30 minutes to merge nova scheduler changes
* 10:02 arturo: icinga downtime cloudvirt1009 for 30 minutes to re-create canary VM ([[phab:T242078|T242078]])

=== 2020-01-06 ===
* 13:45 andrewbogott: restarting nova-api and nova-conductor on cloudcontrol1003 and 1004

=== 2020-01-04 ===
* 16:34 arturo: icinga downtime cloudvirt1024 for 2 months because hardware errors ([[phab:T241884|T241884]])

=== 2019-12-31 ===
* 11:46 andrewbogott: I couldn't!
* 11:40 andrewbogott: restarting cloudservices2002-dev to see if I can reproduce an issue I saw earlier

=== 2019-12-25 ===
* 10:13 arturo: icinga downtime for 30 minutes the whole cloud* lab* fleet to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/560575 (will restart some openstack components)

=== 2019-12-24 ===
* 15:13 arturo: icinga downtime all the lab* fleet for nova password change for 1h
* 14:39 arturo: icinga downtime all the cloud* fleet for nova password change for 1h

=== 2019-12-23 ===
* 11:13 arturo: enable puppet in cloudcontrol1003/1004
* 10:40 arturo: disable puppet in cloudcontrol1003/1004 while doing changes related to python-ldap

=== 2019-12-22 ===
* 23:48 andrewbogott: restarting nova-conductor and nova-api on cloudcontrol1003 and 1004
* 09:45 arturo: cloudvirt1013 is back (did it alone) [[phab:T241313|T241313]]
* 09:37 arturo: cloudvirt1013 is down for good. Apparently powered off. I can't even reach it via iLO

=== 2019-12-20 ===
* 12:43 arturo: icinga downtime cloudmetrics1001 for 128 hours

=== 2019-12-18 ===
* 12:55 arturo: [codfw1dev] created a new subnet neutron object to hold the new CIDR for floating IPs (cloud-codfw1dev-floating - 185.15.57.0/29) [[phab:T239347|T239347]]

=== 2019-12-17 ===
* 07:21 andrewbogott: deploying horizon/train to labweb1001/1002

=== 2019-12-12 ===
* 06:11 arturo: schedule 4h downtime for labstores
* 05:57 arturo: schedule 4h downtime for cloudvirts and other openstack components due to upgrade ops

=== 2019-12-02 ===
* 06:28 andrewbogott: running  nova-manage db sync on eqiad1
* 06:27 andrewbogott: running nova-manage cell_v2 map_cell0 on eqiad1

=== 2019-11-21 ===
* 16:07 jeh: created replica indexes and views for szywiki [[phab:T237373|T237373]]
* 15:48 jeh: creating replica indexes and views for shywiktionary [[phab:T238115|T238115]]
* 15:48 jeh: creating replica indexes and views for gcrwiki [[phab:T238114|T238114]]
* 15:46 jeh: creating replica indexes and views for minwiktionary [[phab:T238522|T238522]]
* 15:36 jeh: creating replica indexes and views for gewikimedia [[phab:T236404|T236404]]

=== 2019-11-18 ===
* 19:27 andrewbogott: repooling labsdb1011
* 18:54 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1011    [[phab:T238480|T238480]]
* 18:44 andrewbogott: depooling labsdb1011 and killing remaining user queries  [[phab:T238480|T238480]]
* 18:42 andrewbogott: repooled labsdb1009 and 1010  [[phab:T238480|T238480]]
* 18:19 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1010    [[phab:T238480|T238480]]
* 18:18 andrewbogott: depooling labsdb1010, killing remaining user queries
* 17:46 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1009    [[phab:T238480|T238480]]
* 17:38 andrewbogott: depooling labsdb1009, killing remaining user queries
* 16:54 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1012   [[phab:T237509|T237509]]

=== 2019-11-15 ===
* 20:04 andrewbogott: repool labdb1011 ([[phab:T237509|T237509]])
* 19:29 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1011
* 19:25 andrewbogott: depooling labsdb1011, killing remaining queries
* 19:25 andrewbogott: repooling labsdb1010
* 18:59 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1012
* 18:57 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1010
* 18:54 andrewbogott: depooling labsdb1010, killing remaining user queries
* 18:54 andrewbogott: depooled labsdb1009, ran maintain-views —clean —all-databases —replace-all, repooled

=== 2019-11-11 ===
* 13:10 arturo: cloudweb2001-dev: disable puppet and redirect stderr in the loadExitNodes.php cron script to prevent cronspam while we investigate the cause of the issue ([[phab:T237971|T237971]])

=== 2019-11-05 ===
* 11:59 arturo: icinga downtime for 1h cloudcontrol1004, cloudnet1003, cloudvirt1017/1020/1022 for PDU operations in the rack [[phab:T227542|T227542]]

=== 2019-11-04 ===
* 21:55 andrewbogott: deleting a ton of wikitech hiera pages that were either no-ops or refer to nonexistent VMs or prefixes

=== 2019-10-31 ===
* 11:01 arturo: icinga-downtimed cloudvirt1030 and cloudservices1003 for 1h due to PDU upgrade operations [[phab:T227543|T227543]]

=== 2019-10-30 ===
* 22:43 jeh: reboot cloud-bootstrapvz-stretch to resolve bad bootstrapvz build

=== 2019-10-29 ===
* 10:52 arturo: icinga downtime cloudvirt1001/1002/1024/1018/1012/1009/1015/1008 for 1h [[phab:T227538|T227538]]

=== 2019-10-25 ===
* 10:45 arturo: icinga downtime toolschecker for 1 to upgrade clouddb1002 mariadb (toolsdb secondary) ([[phab:T236384|T236384]] , [[phab:T236420|T236420]])

=== 2019-10-24 ===
* 12:30 arturo: starting cloudvirt1019, PDU operations ended ([[phab:T227540|T227540]])
* 11:58 arturo: icinga downtime for 2h ([[phab:T227540|T227540]]) cloudvirt1019
* 11:15 arturo: poweroff cloudvirt1019 during the PDU operations ([[phab:T227540|T227540]])
* 11:10 arturo: icinga downtime for 2h ([[phab:T227540|T227540]]) toolschecker
* 10:58 arturo: icinga downtime for 1h ([[phab:T227540|T227540]]) cloudvirt100[3-7], cloudvirt1019, cloudvirt1016, cloudvirt1021, cloudvirt1013, cloudnet1004

=== 2019-10-23 ===
* 09:23 arturo: cloudvirt1026 reboot ended OK
* 09:12 arturo: rebooting cloudvirt1026 for kernel upgrade
* 09:09 arturo: cloudvirt1025 reboot ended OK
* 09:00 arturo: rebooting cloudvirt1025 for kernel upgrade
* 08:51 arturo: icinga downtime cloudvirt1025/1026 for reboots

=== 2019-10-18 ===
* 16:01 arturo: created the `eqiad1.wikimedia.cloud` DNS zone ([[phab:T235846|T235846]])
* 14:27 andrewbogott: deleted a bunch of leaked VMS from earlier today from the admin-monitoring project.   Fullstack leaks due to an api outage, maybe?
* 10:44 arturo: double max_message_size from 40KB to 80KB in the cloud-admin mailing list. A simple email with a couple of quotes can go over the 40KB limit.

=== 2019-10-16 ===
* 21:59 jeh: resync wiki replica tool and user accounts [[phab:T235697|T235697]]
* 09:40 arturo: reboot of cloudvirt1030 went fine
* 09:28 arturo: reboot of cloudvirt1029 went fine
* 09:28 arturo: rebooting cloudvirt1030 for kernel updates
* 09:12 arturo: rebooting cloudvirt1029 for kernel updates
* 09:11 arturo: reboot of cloudvirt1028 went fine
* 09:00 arturo: rebooting cloudvirt1028 for kernel updates
* 08:56 arturo: icinga downtime cloudvirt[1028-1030].eqiad.wmnet for 1h for reboots

=== 2019-10-15 ===
* 13:30 jeh: creating indexes and views for banwiki [[phab:T234770|T234770]]

=== 2019-10-10 ===
* 18:55 bd808: Created indexes and views for nqowiki ([[phab:T230543|T230543]])
* 11:59 arturo: network switch hardware is down affecting cloudvirt1025/1026 ([[phab:T227536|T227536]]) VMs are supposed to be online but unreachable

=== 2019-10-09 ===
* 10:44 arturo: cloudvirt1013 rebooted well
* 10:32 arturo: cloudvirt1013 is rebooting
* 10:32 arturo: cloudvirt1012 rebooted just fine (very slow, 35 VMs)
* 10:21 arturo: cloudvirt1012 is rebooting
* 10:19 arturo: cloudvirt1009 rebooted just fine (very slow though)
* 10:07 arturo: cloudvirt1009 is rebooting
* 10:06 arturo: cloudvirt1008 rebooted just fine (very slow though)
* 09:58 arturo: cloudvirt1008 is rebooting
* 09:52 arturo: icinga downtime toolschecker, paws, etc for 2h, because cloudvirt reboots

=== 2019-10-07 ===
* 14:07 arturo: horizon is disabled for maintenance ([[phab:T212302|T212302]])
* 14:00 arturo: starting scheduled maintenance: upgrading eqiad1 from openstack mitaka to newton

=== 2019-10-02 ===
* 15:23 arturo: codfw1dev renaming net/subnet objects to a more modern naming scheme [[phab:T233665|T233665]]
* 12:49 arturo: codfw1dev delete all floating ip allocations in the deployment for mangling the network config for testing [[phab:T233665|T233665]]
* 12:47 arturo: codfw1dev deleting all VMs in the deployment for mangling the network config for testing [[phab:T233665|T233665]]
* 11:08 arturo: codfw1dev rebooting cloudnet2002-dev and cloudnet2003-dev for testing [[phab:T233665|T233665]]
* 10:31 arturo: codfw1dev: add cloudinstances2b-gw router to the l3 agent in cloudnet2003-dev
* 09:59 arturo: codfw1dev: cleanup leftover &quot;HA port tenant admin&quot; in neutron (ports from missing servers)
* 09:46 arturo: codfw1dev: cleanup leftover neutron agents

=== 2019-09-30 ===
* 10:21 arturo: we installed ferm in every VM by mistake. Deleting it and forcing a puppet agent run to try to go back to a clean state.
* 09:38 arturo: downtime toolschecker for 24h
* 09:33 arturo: force update ferm cloud-wide (in all VMs) for [[phab:T153468|T153468]]

=== 2019-08-18 ===
* 10:39 arturo: rebooting cloudvirt1023 for new interface names configuration
* 10:34 arturo: downtimed cloudvirt1023 for 2 days

=== 2019-08-05 ===
* 17:17 bd808: Set downtime on gridengine and kubernetes webservice checks in icinga until 2019-09-02 (flaky tests)

=== 2019-07-29 ===
* 20:14 bd808: Restarted maintain-kubeusers on tools-k8s-master-01 ([[phab:T194859|T194859]])

=== 2019-07-25 ===
* 12:32 arturo: eqiad1/glance: debian-9.9-stretch image deprecates debian-9.8-stretch ([[phab:T228983|T228983]])
* 09:59 arturo: (codfw1dev) drop missing glance images ([[phab:T228972|T228972]])
* 09:32 arturo: (codfw1dev) deleting a bunch of VMs that were running in now missing hypervisors
* 09:31 arturo: (codfw1dev) deleting a bunch of VMs in ERROR and SHUTDOWN state
* 09:27 arturo: last log entry refers to the codfw1dev deployment
* 09:27 arturo: cleanup `nova service-list` from old hypervisors (labtest*)
* 09:23 arturo: refreshed nova DB grants in clouddb2001-dev for the codfw1dev deployment
* 08:47 arturo: cleanup the cloud-announce pending emails (spam)

=== 2019-07-23 ===
* 19:43 andrewbogott: restarting rabbitmq-server on cloudcontrol1003 and 1004

=== 2019-07-22 ===
* 23:44 bd808: Restarted maintain-kubeusers on tools-k8s-master-01 ([[phab:T228529|T228529]])

=== 2019-07-11 ===
* 22:07 bd808: Ran `sudo systemctl stop designate_floating_ip_ptr_records_updater.service` on cloudcontrol1003
* 22:01 bd808: `sudo apt-get install python2.7-dbg` on cloudcontrol1003 to debug hung python process
* 21:48 bd808: Ran `sudo systemctl stop designate_floating_ip_ptr_records_updater.service` on cloudcontrol1004

=== 2019-06-25 ===
* 16:05 bstorm_: updated python3.4 to update4 wherever it was installed on Jessie VMs to prevent issues with broken update3.
* 14:56 bstorm_: Updated python 3.4 on the labs-puppetmaster server

=== 2019-06-03 ===
* 15:55 arturo: [[phab:T221769|T221769]] rebooting cloudservices1003 after bootstrapping is apparently completed

=== 2019-05-28 ===
* 21:42 bstorm_: unmounting labstore1003-scratch on all cloud clients
* 18:14 bstorm_: [[phab:T209527|T209527]] switched mounts from labstore1003 to cloudstore1008 for scratch

=== 2019-05-20 ===
* 17:25 arturo: [[phab:T223923|T223923]] dropped compat-network config from /etc/network/interfaces in eqiad1/codfw1dev neutron nodes
* 17:22 arturo: [[phab:T223923|T223923]] dropped br-compat bridges and vlan interfaces (1102 and 2102) in eqiad1/codfw1dev neutron nodes
* 17:07 arturo: [[phab:T223923|T223923]] dropped compat-network configuration from the neutron database in eqiad1
* 16:55 arturo: [[phab:T223923|T223923]] dropped compat-network configuration from the neutron database in codfw1dev

=== 2019-05-15 ===
* 17:00 andrewbogott: touching /root/firstboot_done on all VMs that cumin can reach.  This will prevent firstboot.sh from running a second time if/when any of these are rebooted.  [[phab:T223370|T223370]]

=== 2019-04-26 ===
* 15:51 arturo: andrew updated dns servers for the cloud-instances2-b-eqiad subnet in neutron: 208.80.154.143 and 208.80.154.24

=== 2019-04-25 ===
* 11:14 arturo: [[phab:T221760|T221760]] increased size of conntrack table

=== 2019-04-24 ===
* 12:54 arturo: [[phab:T220051|T220051]] puppet broken in every VM in Cloud VPS, fixing right now

=== 2019-04-22 ===
* 11:14 arturo: create by hand /var/cache/labsaliaser/labs-ip-aliases.json in cloudservices2002-dev ([[phab:T218575|T218575]])

=== 2019-04-16 ===
* 22:55 bd808: cloudcontrol2003-dev: added `exit 0` to /etc/cron.hourly/keystone to stop cron spam on partially configured cluster
* 12:08 arturo: rebooting cloudvirt200[123]-dev because deep changes in config
* 11:27 arturo: [[phab:T219626|T219626]] add DB grants for neutron and glnace to clouddb2001-dev (codfw1dev)
* 10:37 arturo: [[phab:T219626|T219626]] replace 208.80.153.75 with 208.80.153.59 in the clouddb2001-dev database (codfw1dev deployment)
* 10:30 arturo: [[phab:T219626|T219626]] replace labtestcontrol2003 with cloudcontrol2001-dev in the clouddb2001-dev database (codfw1dev deployment)

=== 2019-04-15 ===
* 13:08 arturo: [[phab:T219626|T219626]] add DB grants for keystone/nova/nova_api to clouddb2001-dev (codfw1dev)

=== 2019-04-13 ===
* 18:25 bd808: Restarted nova-compute service on cloudvirt1015 ([[phab:T220853|T220853]])

=== 2019-04-11 ===
* 12:00 arturo: [[phab:T151704|T151704]] deploying oidentd to cloudnet1xxx servers

=== 2019-04-02 ===
* 19:52 andrewbogott: installed new base Stretch image.  Updated packages, and runs apt-get dist-upgrade on first boot.

=== 2019-03-29 ===
* 14:34 andrewbogott: moving tools-static.wmflabs.org to point to tools-static-13 in eqiad1-r
* 00:00 bstorm_: [[phab:T193264|T193264]] Added osm.db.svc.eqiad.wmflabs to cloud DNS

=== 2019-03-25 ===
* 00:40 bd808: Restarted maintain-dbusers on labstore1004. Process hung up on failed LDAP connection.

=== 2019-03-21 ===
* 19:32 andrewbogott: restarting keystone on cloudcontrol1003

=== 2019-03-15 ===
* 16:00 gtirloni: increased nscd cache size ([[phab:T217280|T217280]])

=== 2019-03-14 ===
* 19:04 gtirloni: bstorm started nfsd on labstore1006 ([[phab:T218341|T218341]])
* 16:42 gtirloni: published new debian-9.8 image ([[phab:T218314|T218314]])

=== 2019-03-04 ===
* 19:37 bstorm_: umounted /mnt/nfs/dumps-labstore1006.wikimedia.org across all VPS projects for [[phab:T217473|T217473]]

=== 2019-02-26 ===
* 12:46 gtirloni: shutdown toolsbeta-sgegrid-master (cronspam)

=== 2019-02-25 ===
* 10:32 gtirloni: restarted nfsd on labstore1004

=== 2019-02-21 ===
* 09:09 gtirloni: restarted uwsgi-labspuppetbackend.service on labpuppetmaster1001
* 07:42 gtirloni: created project cloudstore
* 07:36 gtirloni: deleted wmcs-nfs project

=== 2019-02-20 ===
* 21:58 andrewbogott: silencing shinken and disabling puppet on shinken-02 for now

=== 2019-02-19 ===
* 12:00 gtirloni: added nagios@icinga2001.wikimedia.org to cloud-admin-feed@ allowed senders

=== 2019-02-18 ===
* 20:21 gtirloni: downtimed cloudvirt1020
* 20:12 gtirloni: ran `labs-ip-alias-dump.py` on cloudservices/labservices servers

=== 2019-02-15 ===
* 13:10 arturo: [[phab:T216239|T216239]] labvirt1019 has been drained
* 12:22 arturo: [[phab:T216239|T216239]] draining labvirt1009 with a command like this: `root@cloudcontrol1004:~# wmcs-cold-migrate --region eqiad --nova-db nova 2c0cf363-c7c3-42ad-94bd-{{Gerrit|e586f2492321}} labvirt1001`
* 12:02 arturo: more nova service cleanups in the database (labvirts that were reallocated to eqiad1)
* 11:34 arturo: [[phab:T216190|T216190]] cleanup from nova database `nova service-delete 35`
* 03:50 andrewbogott: updated VPS base images for Jessie and Stretch, now featuring Stretch 9.7

=== 2019-02-11 ===
* 18:13 gtirloni: cleaned old metrics data in labmon1001 [[phab:T215417|T215417]]
* 15:28 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1011
* 14:18 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1010

=== 2019-02-08 ===
* 14:56 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1009

=== 2019-02-06 ===
* 11:47 gtirloni: downtimed labmon100{1,2} [[phab:T215399|T215399]]
* 00:17 bstorm_: [[phab:T214106|T214106]] deleted bstorm-test2 project to clean up

=== 2019-02-05 ===
* 10:48 arturo: labmon1001 is now part of the 'eqiad1-r' region

=== 2019-02-01 ===
* 09:54 arturo: moving canary1015-01 VM instance from cloudvirt1024 back to cloudvirt1015

=== 2019-01-31 ===
* 12:44 arturo: [[phab:T215012|T215012]] depooling cloudvirt1015 and migrating all VMs to cloudvirt1024

=== 2019-01-25 ===
* 20:11 gtirloni: deleted project yandex-proxy [[phab:T212306|T212306]]
* 20:11 gtirloni: deleted project [[phab:T212306|T212306]]

=== 2019-01-24 ===
* 11:50 arturo: [[phab:T213925|T213925]] modify subnet cloud-instances-transport1-b-eqiad1 to avoid floating IP allocations from here
* 11:07 arturo: [[phab:T214299|T214299]] failover cloudnet1003 to cloudnet1004
* 10:03 arturo: [[phab:T214299|T214299]] reimage cloudnet1004 to debian stretch
* 09:51 arturo: [[phab:T214299|T214299]] failover cloudnet1004 to cloudnet1003

=== 2019-01-22 ===
* 19:19 arturo: [[phab:T214299|T214299]] stretch cloudnet1003 is apparently all set
* 18:40 arturo: [[phab:T214299|T214299]] manually delete from neutron agents from cloudnet1003 (must be added again after reimage, with new uuids)
* 18:37 arturo: [[phab:T214299|T214299]] reimaging cloudnet1003 as debian stretch
* 17:35 jbond42: starting roll out of apt package updates to
* 14:41 gtirloni: [[phab:T214369|T214369]] deployed new jessie and stretch VM images

=== 2019-01-21 ===
* 18:29 gtirloni: installed libguestfs-tools on cloudvirt1021

=== 2019-01-16 ===
* 14:21 andrewbogott: stopping old VPS proxies in eqiad — [[phab:T213540|T213540]]

=== 2019-01-15 ===
* 14:20 andrewbogott: changing tools.wmflabs.org to point to tools-proxy-03 in eqiad1

=== 2019-01-13 ===
* 20:00 andrewbogott: VPS proxies are now running in eqiad1 on proxy-01.  Old VMs will wait a bit for deletion.  [[phab:T213540|T213540]]
* 19:12 andrewbogott: moving the VPS proxy API backend to proxy-01.project-proxy.eqiad.wmflabs, as per [[phab:T213540|T213540]]
* 17:11 andrewbogott: moving all VPS dynamic proxies to proxy-eqiad1.wmflabs.org aka proxy-01.project-proxy.eqiad.wmflabs, as per [[phab:T213540|T213540]]

=== 2019-01-09 ===
* 22:21 bd808: neutron quota-update --tenant-id tools --port 256

=== 2019-01-08 ===
* 18:59 bd808: Definately did NOT delete uid=novaadmin,ou=people,dc=wikimedia,dc=org
* 18:59 bd808: Deleted LDAP user uid=neutron,ou=people,dc=wikimedia,dc=org
* 18:58 bd808: Deleted LDAP user uid=novaadmin,ou=people,dc=wikimedia,dc=org

=== 2019-01-06 ===
* 22:03 bd808: Set floatingip quota of 60 for tools project in eqiad1-r region ([[phab:T212360|T212360]])

=== 2018-12-20 ===
* 17:10 arturo: [[phab:T207663|T207663]] renumbered transport network in eqiad1

=== 2018-12-05 ===
* 17:59 arturo: [[phab:T207663|T207663]] changed labtestn transport network addressing from private to public

=== 2018-12-03 ===
* 13:25 arturo: [[phab:T202886|T202886]] create again PTR records after dnsleak.py fix

=== 2018-11-30 ===
* 14:08 arturo: running dns leaks cleanup `root@cloudcontrol1003:~# /root/novastats/dnsleaks.py --delete`

=== 2018-11-28 ===
* 17:33 gtirloni: deleted contintcloud project ([[phab:T209644|T209644]])

=== 2018-11-27 ===
* 13:32 gtirloni: enabled DRBD stats collection on labstore100[4-5] [[phab:T208446|T208446]]

=== 2018-11-22 ===
* 07:12 gtirloni: deployed new debian-9.6-stretch image

=== 2018-11-21 ===
* 10:48 arturo: re-created compat-net as not shared in labtestn to test stuff related to [[phab:T209954|T209954]]

=== 2018-11-16 ===
* 12:43 gtirloni: armed keyholder on labpuppetmaster1001/1002 after reboots
* 12:08 gtirloni: rebooted labpuppetmaster1001 ([[phab:T207377|T207377]])
* 11:57 gtirloni: rebooted labpuppetmaster1002 ([[phab:T207377|T207377]])

=== 2018-11-14 ===
* 17:19 gtirloni: added cloudvirt1016 to scheduler pool ([[phab:T209426|T209426]])
* 15:41 gtirloni: reimaging labvirt1016 as cloudvirt1016
* 15:14 gtirloni: reset-failed systemd unit nova-scheduler on cloudcontrol1004
* 13:52 gtirloni: rebooted labservices1002 after package upgrades ([[phab:T207377|T207377]])
* 13:23 gtirloni: rebooted labstore2004 after package upgrades ([[phab:T207377|T207377]])
* 13:20 gtirloni: rebooted labstore2003 after package upgrades ([[phab:T207377|T207377]])
* 13:20 gtirloni: rebooted labstore2001/labstore2003 after package upgrades ([[phab:T207377|T207377]])
* 12:08 gtirloni: rebooted labnet1002 after package upgrades
* 12:01 gtirloni: rebooted labmon1002 after package upgrades
* 11:41 gtirloni: rebooted labcontrol1002 after package upgrades
* 11:15 gtirloni: rebooted cloudcontrol1004 after package upgrades

=== 2018-11-09 ===
* 18:17 gtirloni: restarted neutron-linuxbridge-agent on cloudvirt1018/1023

=== 2018-11-08 ===
* 11:00 gtirloni: Added novaproxy-02 to $CACHES
* 10:50 gtirloni: Added cloudvirt1017 to eqiad1 region

=== 2018-11-07 ===
* 13:49 arturo: [[phab:T208733|T208733]] moving labvirt1017 from main deployment to eqiad1 and renaming it to cloudvirt1017

=== 2018-10-22 ===
* 16:24 arturo: [[phab:T206261|T206261]] another update to dmz_cidr in eqiad1
* 10:26 arturo: change again in dmz_cidr in eqiad1: VMs will connect between them without NAT even when using floating IPs ([[phab:T206261|T206261]])

=== 2018-10-19 ===
* 12:02 arturo: revert change in dmz_cidr in eqiad1 for now ([[phab:T206261|T206261]])
* 11:16 arturo: change in dmz_cidr in eqiad1: VMs will connect between them without NAT even when using floating IPs ([[phab:T206261|T206261]])
* 10:14 arturo: we have new virt servers in the eqiad1 deployment since past week and this week: cloudvirt1018, cloudvirt1023, cloudvirt1024

=== 2018-09-26 ===
* 10:40 arturo: [[phab:T205524|T205524]] all sorts of restarts in all neutron daemons
* 10:20 arturo: [[phab:T205524|T205524]] stop/start all neutron agents in cloudnet1003.eqiad.wmnet
* 10:13 arturo: [[phab:T205524|T205524]] restart all agents in cloudnet1004.eqiad.wmnet
* 10:10 arturo: restart neutron-server in cloudcontrol1003, investigating [[phab:T205524|T205524]]

=== 2018-09-24 ===
* 10:57 arturo: try to increase floating ip allocation pool in eqiad1. Of 185.15.56.0/25 we are using only 185.15.56.10-185.15.56.31, I don't know why. Let's use 185.15.56.2-185.15.56.126

=== 2018-09-21 ===
* 17:18 bd808: Running `sudo maintain-meta_p --all-databases --purge` across labsdb10(09{{!}}10{{!}}11) for [[phab:T201890|T201890]]

=== 2018-09-17 ===
* 22:08 bd808: Granted gtirloni project roles of admin, projectadmin, and user

=== 2018-09-12 ===
* 11:20 arturo: [[phab:T202636|T202636]] distributing default routes using classless-static-route for all VMs in main/labtest (dnsmasq/nova-network)

=== 2018-09-11 ===
* 16:52 arturo: again, restarted nova-network after killing all dnsmasq procs in labnet1001 for [[phab:T202636|T202636]]
* 16:08 arturo: restarted nova-network after killing all dnsmasq procs in labnet1001 for [[phab:T202636|T202636]]
* 10:53 arturo: [[phab:T202636|T202636]] creating all the compat-network configuration in neutron
* 10:36 arturo: [[phab:T202636|T202636]] creating br-compat bridge in eqiad1 for the compat network
* 10:33 arturo: [[phab:T202636|T202636]] manually reserve 10.68.23.253 (in nova-network)

=== 2018-09-10 ===
* 22:46 andrewbogott: deleting all VMs on labvirt1019 and 1020 as prep for [[phab:T204003|T204003]]

=== 2018-08-30 ===
* 15:46 andrewbogott: restarting rabbitmq-server on cloudcontrol1003
* 13:07 arturo: [[phab:T202636|T202636]] internal network routing now exists in labtest/labtestn for VM to communicate with each other

=== 2018-08-28 ===
* 11:04 arturo: [[phab:T202549|T202549]] eqiad1 databases are all now running in m5-master. Mysql has been cleaned from cloudcontrol100[3,4]

=== 2018-08-23 ===
* 16:17 arturo: [[phab:T188589|T188589]] bstorm_ merged patch to reduce nova DB connection usage
* 13:15 arturo: [[phab:T202115|T202115]] `root@cloudcontrol1003:~# neutron subnet-update --allocation-pool start=10.64.22.4,end=10.64.22.4 e4fb2771-a361-4add-ac4e-280cc300c59f`
* 13:10 arturo: [[phab:T202115|T202115]] (was `{&quot;start&quot;: &quot;10.64.22.2&quot;, &quot;end&quot;: &quot;10.64.22.254&quot;}` )
* 13:08 arturo: [[phab:T202115|T202115]] `root@cloudcontrol1003:~# neutron subnet-update --allocation-pool start=10.64.22.254,end=10.64.22.254 e4fb2771-a361-4add-ac4e-280cc300c59f`

=== 2018-08-22 ===
* 15:28 arturo: cleanup local glance,keystone databases in cloudcontrol1003.wikimedia.org (already in m5-master)
* 15:27 arturo: cleanup local keystone database in cloudcontrol1003.wikimedia.org (already in m5-master)

=== 2018-08-21 ===
* 15:39 andrewbogott: initial test message
* 10:31 arturo: eqiad1 remove leftover port for HA on labnet1004
* 10:15 arturo: test

=== 2018-05-07 ===
* 18:07 bstorm_: stopped the toolhistory job because it is totally broken and fills /tmp.

=== 2018-02-09 ===
* 00:55 bd808: Added Arturo Borrero Gonzalez and Bstorm as project members
* 00:54 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])

{{SAL|Project Name=admin}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>m9llsp68kwg9f0nhxh6dhyd01jk5bu9</sha1>
    </revision>
    <revision>
      <id>2309615</id>
      <parentid>2309614</parentid>
      <timestamp>2025-06-07T18:49:39Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova</comment>
      <origin>2309615</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1268507" sha1="kvsg8yzbyuu6uopkchnuqgiobnmjo4w" xml:space="preserve">=== 2025-06-07 ===
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova

=== 2025-06-06 ===
* 20:59 andrewbogott: restarting all designate services on all cloudcontrols in eqiad1

=== 2025-06-05 ===
* 20:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,octavia
* 15:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:52 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2025-06-03 ===
* 22:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 20:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:44 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:43 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 17:50 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:36 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-06-02 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 21:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237
* 21:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242
* 17:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 15:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 00:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T394333|T394333]])

=== 2025-06-01 ===
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T394333|T394333]])

=== 2025-05-31 ===
* 23:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services ([[phab:T395742|T395742]])
* 23:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services ([[phab:T395742|T395742]])
* 23:38 andrewbogott: failing over from cloudnet1005 to 1006 in hopes of unsticking [[phab:T395742|T395742]]
* 23:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,neutron ([[phab:T395742|T395742]])
* 23:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,neutron ([[phab:T395742|T395742]])

=== 2025-05-30 ===
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-28 ===
* 22:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1076.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1076.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1075.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1075.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1074.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1074.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1073.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1073.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1072.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1072.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1071.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1071.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1070.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1070.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1069.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1069.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1068.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1068.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudrabbot1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbot1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,designate
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate

=== 2025-05-26 ===
* 13:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet2006-dev.codfw.wmnet
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet2006-dev.codfw.wmnet
* 13:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet2005-dev.codfw.wmnet
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet2005-dev.codfw.wmnet
* 12:54 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=99)
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.roll_reboot_cloudnets
* 12:49 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=99)
* 12:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.roll_reboot_cloudnets
* 12:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::control&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::control&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 07:52 taavi: add new /dev/sdb back to software raid after it was replaced in cloudcephmon1004

=== 2025-05-23 ===
* 09:00 dhinus: failover dumps_dist_active_vps to clouddumps1001 ([[phab:T383723|T383723]])

=== 2025-05-20 ===
* 19:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 15:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 14:57 andrewbogott: resetting eqiad1 rabbitmq in an attempt to resolve [[phab:T394790|T394790]]
* 03:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 03:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 02:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 00:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 00:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])

=== 2025-05-19 ===
* 22:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 22:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 22:43 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for service: project,nova
* 22:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova
* 22:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:26 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for service: project,nova
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 18:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T394727|T394727]])
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1075.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1075.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1072.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1072.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1076.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1076.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1073.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1073.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-15 ===
* 15:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 14:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 13:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-14 ===
* 17:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 17:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/231
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/231
* 12:59 taavi: powercycle unresponsive cloudnet2006-dev

=== 2025-05-12 ===
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 08:17 taavi: powercycle clouservices2005-dev.codfw.wmnet
* 03:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 03:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 02:36 andrewbogott: rebooting cloudnet2005-dev from mgmt  -- ssh is failing and the console shows a user prompt but not a password prompt.

=== 2025-05-11 ===
* 13:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova

=== 2025-05-07 ===
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 19:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:19 andrewbogott: upgrading codfw1dev to version 'epoxy' [[phab:T390914|T390914]]
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T390914|T390914]])
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T390914|T390914]])
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum,heat
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum,heat
* 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum,heat
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum,heat
* 16:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum
* 16:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,nova
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 13:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,neutron
* 13:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,neutron
* 13:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,neutron
* 13:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,neutron
* 12:48 taavi: updating all security group rules referencing old 172.16.0.0/21 subnet to reference new ip space instead ([[phab:T379175|T379175]])
* 10:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-05-06 ===
* 03:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)

=== 2025-05-05 ===
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 03:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 00:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 00:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 00:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-05-04 ===
* 23:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:53 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:41 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2025-05-03 ===
* 02:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])

=== 2025-05-02 ===
* 23:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 21:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 16:20 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=97) ([[phab:T393196|T393196]])
* 16:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 16:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 16:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrewbogott: sudo cumin --force O&lt;nowiki&gt;{&lt;/nowiki&gt;*&lt;nowiki&gt;}&lt;/nowiki&gt; &quot;dpkg --list {{!}} grep puppetserver &amp;&amp; systemctl restart puppetserver.service&quot; # work around a package update that is causing some puppetservers to error out

=== 2025-04-29 ===
* 13:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225
* 12:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225
* 11:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:55 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223
* 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223
* 07:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 07:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221

=== 2025-04-28 ===
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:21 taavi: migrating opentofu managed default security group rules [[phab:T392799|T392799]]
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-26 ===
* 07:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0) ([[phab:T390134|T390134]])
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance ([[phab:T390134|T390134]])
* 07:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T390134|T390134]])
* 07:47 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T390134|T390134]])

=== 2025-04-25 ===
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218
* 12:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 04:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T390134|T390134]])

=== 2025-04-24 ===
* 17:25 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T390134|T390134]])
* 13:29 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for service: project,designate
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate

=== 2025-04-23 ===
* 21:45 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 21:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:45 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:21 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 19:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:32 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:29 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:29 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:17 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:27 arturo: enabling IPv6 dualstack on neutron virtual router ([[phab:T380174|T380174]])
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:14 arturo: enable IPv6 on cloudgw ([[phab:T380174|T380174]]) -- includes server reboot

=== 2025-04-22 ===
* 04:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)

=== 2025-04-21 ===
* 23:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 23:34 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 23:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 23:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 22:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:09 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 22:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 16:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 01:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 01:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-16 ===
* 16:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 arturo: merging network change in neutron https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/198
* 10:10 dcaro: upgrade spicerack on cloudcumin2001 to 10.1.0
* 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 10:07 dcaro: upgrade spicerack on cloudcumin1001 to 10.1.0

=== 2025-04-15 ===
* 15:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:38 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:31 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:30 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-14 ===
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 17:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:50 andrewbogott: granting 'tofuadmin' user inherited 'member' role in all projects, this should fix some policy mishaps
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-13 ===
* 22:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 22:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-11 ===
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-10 ===
* 12:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:05 arturo: [codfw1dev] root@cloudcontrol2004-dev:~# wmcs-makedomain --project testlabs --domain testlabs.codfw1dev.wmcloud.org --orig-project cloudinfra-codfw1dev ([[phab:T391325|T391325]])

=== 2025-04-09 ===
* 23:23 bd808: Rebooting tools-sgebastion-10 (login-buster.toolforge.org) for high load/unresponsive NFS mounts ([[phab:T391538|T391538]])
* 10:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 08:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 04:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 03:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:30 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 03:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services

=== 2025-04-08 ===
* 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet'
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet'
* 22:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1011.eqiad.wmnet'
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet'
* 11:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2025-04-07 ===
* 23:43 bd808: `sudo service maintain-dbusers restart` on cloudcontrol1007 after reports of missing replica.my.cnf and finding the journal for the service empty.
* 15:30 arturo: [codfw1dev] testlabs create a bunch of VMs by hand, like `networktests-vlan-legacy-floating` [[phab:T380728|T380728]]
* 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:23 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-06 ===
* 02:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 02:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-04 ===
* 15:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 05:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 05:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:10 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for all services
* 05:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:04 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for all services
* 05:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-03 ===
* 22:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 20:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 20:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 20:11 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:29 taavi: run wmcs-wikireplica-dns to create transitional x3 CNAMEs [[phab:T390954|T390954]]

=== 2025-04-02 ===
* 20:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup2004.codfw.wmnet' ([[phab:T381499|T381499]])
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.unset_maintenance (exit_code=0) ({{Gerrit|1133432}})
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.unset_maintenance ({{Gerrit|1133432}})
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:22 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:26 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T381499|T381499]])
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T381499|T381499]])
* 09:53 dhinus: systemctl restart maintain-dbusers.service (attempting to fix a weird issue)

=== 2025-04-01 ===
* 21:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate
* 21:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 21:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 21:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 21:00 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-31 ===
* 15:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 15:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 15:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 15:53 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2025-03-27 ===
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 14:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 12:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T390134|T390134]])

=== 2025-03-26 ===
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-25 ===
* 14:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-24 ===
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-22 ===
* 03:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,designate
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate

=== 2025-03-19 ===
* 14:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,designate
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 12:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,task_id,no_dologmsg,cluster_name,all_services,nova,glance,keystone,cinder,neutron,trove,magnum,heat,swift,designate,filter_nodes
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,task_id,no_dologmsg,cluster_name,all_services,nova,glance,keystone,cinder,neutron,trove,magnum,heat,swift,designate,filter_nodes
* 06:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 06:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 06:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 06:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet'
* 06:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 06:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet'
* 06:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet'
* 06:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 06:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet'
* 06:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 06:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 05:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet'
* 05:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 05:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 05:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 05:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 04:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 04:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 04:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet'
* 04:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 04:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 04:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet'
* 04:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet'
* 04:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet'
* 04:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet'
* 04:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet'
* 04:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 03:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T381499|T381499]])
* 02:12 andrewbogott: upgrading codfw1dev to openstack 'dalmation'  [[phab:T381499|T381499]]
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T381499|T381499]])

=== 2025-03-14 ===
* 13:36 volans: installed cumin v5.1.1 on cloudcumin* hosts

=== 2025-03-05 ===
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 00:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 00:43 andrewbogott: restarting all openstack services in hopes of gettting dns unstuck
* 00:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services

=== 2025-03-04 ===
* 08:52 arturo: [[phab:T387828|T387828]] depooled galera on cloudcontrol1005

=== 2025-03-01 ===
* 19:35 andrewbogott: installed new bookworm base images in eqiad1
* 19:35 andrewbogott: installed new bookworm and bullseye base images in codfw1dev

=== 2025-02-28 ===
* 15:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-27 ===
* 15:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-24 ===
* 00:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2025-02-23 ===
* 21:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-21 ===
* 12:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-20 ===
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T386083|T386083]])
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T386083|T386083]])

=== 2025-02-19 ===
* 13:26 arturo: manual failover of cloudgw1004 to cloudgw1003 [[phab:T382356|T382356]]
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-18 ===
* 12:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-02-13 ===
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-11 ===
* 11:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T386083|T386083]])
* 11:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T386083|T386083]])
* 11:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047' ([[phab:T386083|T386083]])
* 11:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047' ([[phab:T386083|T386083]])

=== 2025-02-07 ===
* 13:52 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet1005.eqiad.wmnet ([[phab:T384946|T384946]])
* 13:48 root@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet1005.eqiad.wmnet ([[phab:T384946|T384946]])
* 02:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 01:48 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2025-02-06 ===
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:45 andrewbogott: cold-migrating all remaining VMs in [[phab:T385264|T385264]] except for 'integration' and 'tools' VMs

=== 2025-02-05 ===
* 19:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 19:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2025-02-04 ===
* 12:48 arturo: replacing cloudgw1002 with cloudgw1004 - [[phab:T382356|T382356]]
* 09:22 arturo: fleet-wide restart of puppetservers [[phab:T385553|T385553]]

=== 2025-02-03 ===
* 13:42 andrewbogott: rebooting proxy-04.project-proxy for the ceph OSD mishap

=== 2025-02-02 ===
* 17:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 17:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2025-01-31 ===
* 01:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-01-30 ===
* 21:39 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 21:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:33 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 12:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1031.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1031.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 03:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 02:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 02:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 00:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 00:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)

=== 2025-01-29 ===
* 23:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 23:43 andrewbogott: resetting rabbitmq in eqiad1 in hopes that it will resolve mysterious openstack misbehaviors
* 20:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2009-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2009-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1007.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1007.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1005.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1005.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-28 ===
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:51 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 13:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 13:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 13:02 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T348643|T348643]])
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 00:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 00:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T348643|T348643]])
* 00:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2025-01-27 ===
* 21:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:52 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 14:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-24 ===
* 13:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node

=== 2025-01-23 ===
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 20:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 15:49 dhinus: cumin 'P:base::cloud_production' 'rm /var/lib/prometheus/node.d/kernel-panic.prom' [[phab:T382961|T382961]]
* 15:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-22 ===
* 14:52 dhinus: cloudcumin[12]001 upgrade spicerack from 8.15.2 to 9.1.0

=== 2025-01-21 ===
* 13:42 andrewbogott: migrating/rebooting VMs as per earlier email,  [[phab:T383583|T383583]]

=== 2025-01-20 ===
* 14:23 dhinus: cumin upgraded form 4.2.0 to 5.0.0 on cloudcumin[12]001. patch [[phab:T346453|T346453]] reapplied.

=== 2025-01-14 ===
* 21:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T383583|T383583]])
* 21:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T383583|T383583]])
* 20:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T383583|T383583]])
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T383583|T383583]])
* 20:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T383583|T383583]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T383583|T383583]])
* 18:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.roll_restart_osd_daemons (exit_code=0)

=== 2025-01-13 ===
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.roll_restart_osd_daemons

=== 2025-01-10 ===
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-01-09 ===
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2025-01-08 ===
* 14:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 13:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2025-01-03 ===
* 21:01 bd808: `sudo service maintain-dbusers restart` on cloudcontrol1005. Report of missing replica.my.cnf and journalctl output empty due to log rotation. ([[phab:T382962|T382962]])

=== 2024-12-18 ===
* 15:33 arturo: cloudgw failover for [[phab:T382220|T382220]]

=== 2024-12-04 ===
* 17:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 17:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 15:09 andrewbogott: rebooted cloudinfra-cloudvps-puppetserver-1, it's so busy that it's unresponsive
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])

=== 2024-11-27 ===
* 13:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet'
* 13:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'

=== 2024-11-26 ===
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 15:16 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:15 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:15 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:15 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:14 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/145
* 15:13 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/145
* 15:12 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:12 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:31 dcaro: added cloudcephmon1004 to the ceph mon pool
* 12:41 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 05:40 andrewbogott: rebooting tools-sgebastion-10.tools.eqiad1.wikimedia.cloud to get NFS things remounted
* 04:56 andrewbogott: 'systemctl restart nfs-server' on tools-nfs-2.tools.eqiad1.wikimedia.cloud

=== 2024-11-25 ===
* 14:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:40 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 13:55 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:28 arturo: create IPv6 networks in eqiad1 ([[phab:T380174|T380174]]), then reverted because network outage
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-23 ===
* 14:58 taavi: removed broken records for re-created VMs in .eqiad.wmflabs zone

=== 2024-11-22 ===
* 14:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:35 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:48 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-20 ===
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2024-11-19 ===
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:43 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:43 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:37 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:35 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 11:39 arturo: [codfw1dev] performing rabbit full reset [[phab:T380208|T380208]]
* 10:26 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:24 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 10:24 arturo: [codfw1dev] restart rabbitmq and nova/neutron services for [[phab:T380208|T380208]]

=== 2024-11-16 ===
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 07:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-15 ===
* 19:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:10 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:30 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:29 arturo: [codfw1dev] restart rabbitmq and designate
* 16:29 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-14 ===
* 09:52 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:48 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-13 ===
* 13:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/120
* 13:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/120

=== 2024-11-11 ===
* 14:49 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-11-08 ===
* 16:47 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:45 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 16:42 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:42 arturo: [codfw1dev] restart all nova services and rabbitmq out of despair
* 16:42 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:45 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:44 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:33 aborrero@cloudcumin2001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 13:32 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:28 arturo: [codfw1dev] restart rabbitmq, openstack services logs show connection errors

=== 2024-11-07 ===
* 17:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 17:42 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-06 ===
* 13:09 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 11:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/116
* 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/116
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-05 ===
* 10:24 arturo: [codfw1dev] disable puppet and make changes for testing [[phab:T378192|T378192]]

=== 2024-11-04 ===
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:09 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:08 aborrero@cloudcumin2001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:08 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 14:08 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:08 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:52 arturo: [codfw1dev] live-hack cloudlb2001-dev and cloudcontrol2004-dev for [[phab:T378192|T378192]]
* 13:52 arturo: [codfw1dev] restart rabbitmq
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-29 ===
* 15:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-28 ===
* 17:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:00 arturo: [codfw1dev] restarting rabbitmq, misbehaving
* 17:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 dhinus: apt full-upgrade and reboot for cloudcumin*
* 16:21 dhinus: upgrade spicerack from 8.8.0 to 8.15.1 on cloudcumin*

=== 2024-10-24 ===
* 15:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-22 ===
* 15:02 taavi: recover access to User:Labslogbot [[phab:T376220|T376220]]

=== 2024-10-21 ===
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:06 arturo: [codfw1dev] restart rabbitmq, tofu shows error talking to the designate API

=== 2024-10-16 ===
* 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-15 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:38 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:33 arturo: cloudgw maintenance, firewall change for [[phab:T374714|T374714]]
* 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-11 ===
* 15:31 arturo: cloudgw maintenance firewall change  [[phab:T374716|T374716]]
* 09:51 arturo: cloudgw network maintenance related to [[phab:T376879|T376879]]
* 08:33 arturo: [codfw1dev] reboot cloudgw2002-dev/2003-dev because network connectivity issues

=== 2024-10-10 ===
* 12:04 arturo: manual network failover in cloudgw because maintenance related to [[phab:T376879|T376879]]
* 10:40 dhinus: cumin 'cloudrabbit*' 'systemctl restart rabbitmq-server' [[phab:T376802|T376802]]
* 09:19 arturo: [codfw1dev] enable IPv6 on cloudgw ([[phab:T374716|T374716]])

=== 2024-10-09 ===
* 14:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:20 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:19 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:19 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:16 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:16 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:58 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:58 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:55 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:55 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:54 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:53 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:53 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95
* 10:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2024-10-08 ===
* 11:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-07 ===
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-04 ===
* 15:12 arturo: cloudservice1005/1006: enable puppet and restore /etc/powerdns/recursor.conf to non-debug mode ([[phab:T374830|T374830]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:32 arturo: cloudservice1005/1006: disable puppet and set `quiet=no` in /etc/powerdns/recursor.conf ([[phab:T374830|T374830]])

=== 2024-10-03 ===
* 14:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:54 arturo: [codfw1dev] delete default security group rule list, now tracking them via tofu-infra
* 14:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:51 arturo: delete default security group rule list, now tracking them via tofu-infra
* 14:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-02 ===
* 15:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:21 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:16 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:15 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:05 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 09:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:10 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])

=== 2024-10-01 ===
* 15:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-30 ===
* 20:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T372814|T372814]])
* 16:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 16:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 13:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 13:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T372814|T372814]])
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 13:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 11:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 11:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 11:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 11:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 10:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 10:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=99)
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=99)
* 09:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 09:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-27 ===
* 15:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:56 arturo: [codfw1dev] restart rabbitmq again
* 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:15 arturo: [codfw1dev] restart rabbitmq
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:04 arturo: [codfw1dev] enable IPv6 on the neutron virtual router [[phab:T375847|T375847]]

=== 2024-09-26 ===
* 14:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 14:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T372814|T372814]])
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 10:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T372814|T372814]])
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:47 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:45 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-25 ===
* 14:11 arturo: [codfw1dev] start proxy-02 vm on proxy-codfw1dev project, it was in shutoff mode for unknown reasons
* 10:33 arturo: [codfw1dev] cleanup unused security groups ([[phab:T375604|T375604]])
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:48 arturo: [codfw1dev] restart rabbitmq on all cloudcontrol servers
* 09:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:35 arturo: [codfw1dev] deletre a bunch of tests and seemingly unused projects

=== 2024-09-24 ===
* 19:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 16:00 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 14:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T348643|T348643]])
* 14:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 14:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:56 arturo: [codfw1dev] restart rabbitmq-server on all 3 nodes
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack
* 09:10 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_rack (exit_code=97)
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack

=== 2024-09-23 ===
* 19:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_rack (exit_code=99)
* 15:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:42 arturo: put cloudvirt1048 in the network-ovs aggregate [[phab:T364457|T364457]]
* 14:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:37 arturo: [codfw1dev] restart rabbitmq
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:23 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-09-21 ===
* 10:17 dhinus: nova host-evacuate cloudvirt1063 ([[phab:T375223|T375223]])
* 09:59 dhinus: openstack aggregate remove host ceph cloudvirt1063 ([[phab:T375223|T375223]])
* 09:59 dhinus: openstack aggregate add host maintenance cloudvirt1063 ([[phab:T375223|T375223]])
* 09:42 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1063.eqiad.wmnet'
* 09:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1063.eqiad.wmnet'

=== 2024-09-20 ===
* 11:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 10:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 08:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T373740|T373740]])
* 08:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T373740|T373740]])

=== 2024-09-19 ===
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T373740|T373740]])
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T373740|T373740]])
* 15:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) ([[phab:T373740|T373740]])
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance ([[phab:T373740|T373740]])
* 10:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51
* 10:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51
* 09:56 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T374043|T374043]])
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 09:51 arturo: [codfw1dev] play with neutron default security group rules (delete, create them, etc) [[phab:T375111|T375111]]
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50

=== 2024-09-18 ===
* 15:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:21 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 15:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/47
* 11:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/47
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:40 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:39 dcaro: restarted rabbitmq-server on all cloudrabbits
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-09-17 ===
* 21:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T374043|T374043]])
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 16:11 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T374043|T374043]])
* 16:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 15:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46

=== 2024-09-16 ===
* 14:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 11:28 arturo: [codfw1dev] created VM bastion-codfw1dev-04 to replace current bastion -03 ([[phab:T374828|T374828]])

=== 2024-09-12 ===
* 10:51 arturo: merging change to keystone wmf hooks https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071230 ([[phab:T374020|T374020]])

=== 2024-09-11 ===
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 16:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 15:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 15:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/41
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/41
* 11:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 07:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T374467|T374467]])
* 07:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T374467|T374467]])

=== 2024-09-10 ===
* 12:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 11:37 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:42 dcaro: hard-rebooting cloudvirt2004-dev (codfw1dev) having io/hardware issues
* 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 08:45 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-09 ===
* 21:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T373986|T373986]])
* 16:36 dcaro: cleaned up dns leaks
* 15:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/39
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/39
* 15:05 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 14:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:06 arturo: merged change to cloudgw NAT setting https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071189
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:42 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:38 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-06 ===
* 23:18 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T373986|T373986]])
* 18:17 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 13:46 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 07:27 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-05 ===
* 21:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 17:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T374043|T374043]])
* 14:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:19 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 14:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 12:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/36
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/36
* 10:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/35
* 10:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/35
* 09:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/34
* 09:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/34
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/33
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/33
* 09:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:22 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/31
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/31
* 09:16 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 08:46 arturo: [codfw1dev] restart rabbitmq @ codfw1dev [[phab:T374002|T374002]]

=== 2024-09-04 ===
* 21:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 19:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:35 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 12:18 arturo: [codfw1dev] restart rabbitmq-server.service on all 3 cloudcontrols, all nova-compute agents are down complaining about rabbitmq being unreachable

=== 2024-09-02 ===
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 11:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28

=== 2024-08-31 ===
* 13:55 andrewbogott: moving tools-redis-7 off of cloudvirt1048 just in case  [[phab:T373740|T373740]]
* 13:39 andrewbogott: rebooting cloudvirt1048 from mgmt, it seems to have crashed

=== 2024-08-30 ===
* 12:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-29 ===
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-08-25 ===
* 22:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-23 ===
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T369044|T369044]])
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T369044|T369044]])
* 00:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-22 ===
* 23:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=97) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.unset_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.unset_maintenance ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=97) on host 'cloudvirt1039' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1037' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1035' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035' ([[phab:T369044|T369044]])
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1033' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1052' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052' ([[phab:T369044|T369044]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])

=== 2024-08-20 ===
* 03:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)

=== 2024-08-19 ===
* 23:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 23:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 23:16 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-08-17 ===
* 03:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-16 ===
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:20 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T363344|T363344]])
* 05:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:57 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:50 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 01:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:12 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-15 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:27 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 08:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 04:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 02:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-14 ===
* 23:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:05 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-12 ===
* 15:44 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 08:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=0)
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:37 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97) ([[phab:T371878|T371878]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) ([[phab:T371878|T371878]])
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) ([[phab:T371878|T371878]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:17 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97)
* 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_rack

=== 2024-08-09 ===
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 18:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 13:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 13:38 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 13:35 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 13:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 11:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 05:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 05:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 00:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 00:47 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])

=== 2024-08-08 ===
* 23:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet'
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet'
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 18:39 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet'
* 18:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet'
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet'
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet'
* 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet'
* 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet'
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet'
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 16:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet'
* 16:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet'
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudbackup1002-dev.codfw.wmnet'
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.codfw.wmnet'
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet'
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet'
* 16:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2006-dev.codfw.wmnet'
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2006-dev.codfw.wmnet'
* 16:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet'
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 15:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 15:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.wikimedia.org'
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.wikimedia.org'
* 15:51 andrewbogott: upgrading codfw1dev to openstack version caracal https://phabricator.wikimedia.org/T369044
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 15:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 15:49 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T369044|T369044]])
* 15:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 14:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 14:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.wait_for_rebalance
* 13:00 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 12:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 12:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 12:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 11:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T371878|T371878]])
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T371878|T371878]])
* 09:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 09:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 07:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 07:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 06:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 05:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 05:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)

=== 2024-08-07 ===
* 20:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 20:11 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=97)
* 20:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 20:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 20:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 18:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 17:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:56 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=97)
* 16:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node
* 16:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node
* 16:40 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:24 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T371878|T371878]])
* 08:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T371878|T371878]])
* 06:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 03:08 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])

=== 2024-08-06 ===
* 21:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 19:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 18:36 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet'
* 18:21 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1042.eqiad.wmnet'
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet'
* 18:17 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet'
* 18:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet'
* 17:59 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet'
* 17:58 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet'
* 17:45 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet'
* 17:43 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet'
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet'
* 17:32 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet'
* 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1045.eqiad.wmnet'
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet'
* 17:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet'
* 17:21 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet'
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet'
* 17:14 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet'
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet'
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 17:01 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet'
* 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 17:00 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:48 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:47 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 16:47 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:37 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 16:37 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:25 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance

=== 2024-08-01 ===
* 13:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-07-31 ===
* 23:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 23:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 16:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27

=== 2024-07-30 ===
* 11:29 arturo: installing nova security updates ([[phab:T371240|T371240]])

=== 2024-07-29 ===
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:28 arturo: [codfw1dev] restarting rabbitmq-server on all cloudcontrols, nova-compute cannot contact it
* 11:00 arturo: [codfw1dev] installing nova security updates ([[phab:T371240|T371240]])
* 10:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27

=== 2024-07-25 ===
* 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:46 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:45 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:09 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:55 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 11:40 arturo: manually restart maintain-dbusers in cloudcontrol1005 to see if that makes any difference
* 11:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 07:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 07:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23

=== 2024-07-24 ===
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 16:01 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:46 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:43 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:37 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 11:59 arturo: restarted maintain-dbusers.service in cloudcontrol1005, it was stuck doing nothing
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:43 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:42 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24

=== 2024-07-23 ===
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 10:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 09:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 05:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 05:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 05:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 05:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 03:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 03:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 02:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 02:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)

=== 2024-07-22 ===
* 23:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 23:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 21:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 20:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 18:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 15:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:28 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 13:28 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:10 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy

=== 2024-07-21 ===
* 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 09:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 06:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 06:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 03:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 03:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 00:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-20 ===
* 21:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 18:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 18:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 18:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 06:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 06:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 03:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 01:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 00:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-19 ===
* 22:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 19:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 18:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 18:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 06:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 06:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 01:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-18 ===
* 22:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 22:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 19:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 12:31 dhinus: upgrade spicerack from 8.5 to 8.8 on cloudcumin*
* 10:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 07:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 07:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 03:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-17 ===
* 22:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 17:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 07:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 05:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 05:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 02:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 02:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-16 ===
* 23:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 23:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 17:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 11:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 11:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 09:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 09:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-07-15 ===
* 22:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:40 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 20:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 18:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy

=== 2024-07-11 ===
* 13:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 13:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 11:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 11:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 02:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'

=== 2024-07-08 ===
* 17:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T309789|T309789]])
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-07-06 ===
* 14:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-07-05 ===
* 10:33 arturo: aborrero@cloudcephmon1001:~$ sudo ceph osd unset norebalance
* 10:31 arturo: aborrero@cloudcephmon1001:~$ sudo ceph osd unset noin
* 08:56 arturo: installing nova/glance/cinder security updates [[phab:T369138|T369138]]

=== 2024-07-04 ===
* 20:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 20:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:52 dcaro: rebooting cloudcontrol1007 due to systemd-journal service failing to start
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-03 ===
* 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 16:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 12:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-02 ===
* 19:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-01 ===
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 14:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-06-27 ===
* 22:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet'
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1059.eqiad.wmnet'
* 19:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1059.eqiad.wmnet'
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1059.eqiad.wmnet'
* 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet'
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1064.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet'
* 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1067.eqiad.wmnet'
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet'
* 17:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1058.eqiad.wmnet'
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet'
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1066.eqiad.wmnet'
* 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet'
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1065.eqiad.wmnet'
* 17:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet'
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1057.eqiad.wmnet'
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 13:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 12:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet'
* 12:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirtlocal1001.eqiad.wmnet'
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 12:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 10:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 08:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 08:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 07:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 07:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2024-06-26 ===
* 22:32 andrewbogott: disabled all g3.* flavors in eqiad1
* 19:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 17:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 11:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 10:03 dcaro: taking cloudcephosd1006 out of the pool ([[phab:T348643|T348643]])
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 10:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-06-25 ===
* 22:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2003-dev.codfw.wmnet'
* 22:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2003-dev.codfw.wmnet'
* 22:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2002-dev.codfw.wmnet'
* 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2002-dev.codfw.wmnet'
* 22:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codfw.wmnet'
* 22:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2001-dev.codw.wmnet'
* 22:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codw.wmnet'
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2005-dev.codfw.wmnet'
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2004-dev.codfw.wmnet'
* 16:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt2006-dev.codfw.wmnet'
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2006-dev.codfw.wmnet'
* 03:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-24 ===
* 19:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet'
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056.eqiad.wmnet'
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet'
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet'
* 16:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet'
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet'

=== 2024-06-21 ===
* 10:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:35 wmbot~arturo@nostromo: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97)
* 10:35 wmbot~arturo@nostromo: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:35 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:35 wmbot~arturo@nostromo: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 09:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 08:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T368129|T368129]])
* 08:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T368129|T368129]])
* 04:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 4e612eb8-04e1-4541-941d-{{Gerrit|a05519eed60a}}
* 04:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 4e612eb8-04e1-4541-941d-{{Gerrit|a05519eed60a}}

=== 2024-06-20 ===
* 21:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 14789ac1-bc06-4677-9bb0-{{Gerrit|66c16c887427}}
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 14789ac1-bc06-4677-9bb0-{{Gerrit|66c16c887427}}
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet'
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet'
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet'
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1052.eqiad.wmnet'
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet'
* 13:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 614f9c99-86f1-410f-8ef5-{{Gerrit|e33d23215ff5}}
* 13:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 614f9c99-86f1-410f-8ef5-{{Gerrit|e33d23215ff5}}
* 13:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1051.eqiad.wmnet'
* 13:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet'
* 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1050.eqiad.wmnet'
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet'
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049.eqiad.wmnet'
* 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet'
* 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet'
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet'
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet'
* 10:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet'
* 09:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet'
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet'
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1045.eqiad.wmnet'
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 04:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server d0b1d9d5-1aec-4a05-a2d7-{{Gerrit|6d8522a365dc}}
* 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server d0b1d9d5-1aec-4a05-a2d7-{{Gerrit|6d8522a365dc}}
* 04:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 2d2e8925-b50f-483a-82ee-{{Gerrit|e6a1c588e5be}}
* 04:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 2d2e8925-b50f-483a-82ee-{{Gerrit|e6a1c588e5be}}
* 04:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 138be95d-93ad-4a85-9245-{{Gerrit|a0a508711555}}
* 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 138be95d-93ad-4a85-9245-{{Gerrit|a0a508711555}}
* 04:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server a3e945dc-3548-47fa-8ce3-{{Gerrit|bf1426ff3b15}}
* 04:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 0258b810-29af-448a-af5e-{{Gerrit|ed39e19286df}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server a3e945dc-3548-47fa-8ce3-{{Gerrit|bf1426ff3b15}}
* 04:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 37e23659-4516-4bd8-a9be-{{Gerrit|4cc55def5560}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 0258b810-29af-448a-af5e-{{Gerrit|ed39e19286df}}
* 04:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 7da824f3-c9f3-4460-b9b2-{{Gerrit|2a894268a7ca}}
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 37e23659-4516-4bd8-a9be-{{Gerrit|4cc55def5560}}
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 63b82d38-3026-408f-8dcc-{{Gerrit|0ecbd1a3c870}}
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 7da824f3-c9f3-4460-b9b2-{{Gerrit|2a894268a7ca}}
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 7cb371bb-a53a-4e65-a1cf-{{Gerrit|f1a8264a9166}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 63b82d38-3026-408f-8dcc-{{Gerrit|0ecbd1a3c870}}
* 04:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 6616fcbf-a49e-4e03-b735-{{Gerrit|84d31b535c08}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 6616fcbf-a49e-4e03-b735-{{Gerrit|84d31b535c08}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 7cb371bb-a53a-4e65-a1cf-{{Gerrit|f1a8264a9166}}
* 04:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 6f1171db-9d7d-466f-aaa7-{{Gerrit|cb14e1a6af41}}
* 04:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 6f1171db-9d7d-466f-aaa7-{{Gerrit|cb14e1a6af41}}
* 04:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 1dc3fcee-cf27-4351-ad60-{{Gerrit|384478624ea3}}
* 04:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 1dc3fcee-cf27-4351-ad60-{{Gerrit|384478624ea3}}
* 04:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 3df62375-e75d-4068-9c71-{{Gerrit|13519b6bf927}}
* 04:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 3af9b40f-d29d-4216-9b64-{{Gerrit|0ebb10f94c7c}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 3df62375-e75d-4068-9c71-{{Gerrit|13519b6bf927}}
* 04:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server ae9fa949-e4ce-4ffb-ad9f-{{Gerrit|4e5a3812d031}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server ae9fa949-e4ce-4ffb-ad9f-{{Gerrit|4e5a3812d031}}
* 04:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 452dc8d3-6ee0-412c-90ba-{{Gerrit|66f21f9d09c1}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 3af9b40f-d29d-4216-9b64-{{Gerrit|0ebb10f94c7c}}
* 04:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 452dc8d3-6ee0-412c-90ba-{{Gerrit|66f21f9d09c1}}
* 04:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 47b0da1d-e50a-42c1-8cd9-{{Gerrit|dc255cc2f1a3}}
* 04:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 47b0da1d-e50a-42c1-8cd9-{{Gerrit|dc255cc2f1a3}}
* 02:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T368007|T368007]])
* 02:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T368007|T368007]])
* 02:05 andrewbogott: cloudvirt1063 is unresponsive, cycling power from racadm

=== 2024-06-19 ===
* 15:43 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1044.eqiad.wmnet'
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 15:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet'
* 12:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 12:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet'
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet'
* 11:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet'
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1042.eqiad.wmnet'
* 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server f667d3c2-379a-48d7-ad44-{{Gerrit|4f3933bdb871}}
* 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server f667d3c2-379a-48d7-ad44-{{Gerrit|4f3933bdb871}}
* 01:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 71e4296d-039d-4452-9d92-{{Gerrit|69b9f8eb3aba}}
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 71e4296d-039d-4452-9d92-{{Gerrit|69b9f8eb3aba}}
* 01:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server a7725cd2-6162-41a3-8add-{{Gerrit|4dc0668b233b}}
* 01:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server a7725cd2-6162-41a3-8add-{{Gerrit|4dc0668b233b}}
* 01:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}

=== 2024-06-18 ===
* 21:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2024-06-17 ===
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T364457|T364457]])
* 20:03 andrewbogott: repaced ovs hosts in the 'ceph' aggregate
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T364457|T364457]])
* 18:16 andrewbogott: temporarily removing all ovs hosts from the 'ceph' aggregate so the scheduler will stop putting linuxbridge hosts on ovs hosts and breaking them
* 17:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T364457|T364457]])
* 13:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet'

=== 2024-06-14 ===
* 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet'
* 13:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet'

=== 2024-06-13 ===
* 16:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet'
* 13:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet'
* 13:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2003-dev.codfw.wmnet'
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2003-dev.codfw.wmnet'

=== 2024-06-12 ===
* 11:49 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1031.eqiad.wmnet'
* 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet'
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet'
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet'
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet'
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet'

=== 2024-06-11 ===
* 15:55 dcaro: restarting mon service on cloudcephmon1002 to try to release the 2 stuck ops left
* 13:30 taavi: pin all existing eqiad1 flavors to linuxbridge hypervisors [[phab:T364458|T364458]]
* 12:28 taavi: add all existing eqiad1 cloudvirts to new network-linuxbridge aggregate [[phab:T364458|T364458]]
* 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-07 ===
* 19:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-05 ===
* 11:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-06-04 ===
* 15:49 taavi: drop hopefully-unused 68.10.in-addr.arpa. from designate [[phab:T361220|T361220]]

=== 2024-05-30 ===
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-29 ===
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T364984|T364984]])
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T364984|T364984]])
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T364984|T364984]])
* 18:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T364984|T364984]])

=== 2024-05-28 ===
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-21 ===
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 08:18 taavi: stop neutron services on cloudnet1005 [[phab:T364459|T364459]]

=== 2024-05-20 ===
* 14:46 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:46 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:20 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:19 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:18 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:18 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:17 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:17 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:16 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:16 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:15 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:15 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs

=== 2024-05-16 ===
* 08:55 taavi: delete 'monitoring' project https://phabricator.wikimedia.org/T365105

=== 2024-05-15 ===
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T319184|T319184]])
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-05-14 ===
* 19:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-10 ===
* 13:23 andrewbogott: deploying updated 2024.1 Horizon

=== 2024-05-09 ===
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-25 ===
* 21:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T356287|T356287]])
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T356287|T356287]])
* 21:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1033' ([[phab:T356287|T356287]])
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033' ([[phab:T356287|T356287]])
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T356287|T356287]])
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T356287|T356287]])

=== 2024-04-23 ===
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2002-dev.codfw.wmnet'
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2002-dev.codfw.wmnet'

=== 2024-04-18 ===
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2008-dev.codfw.wmnet'
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2008-dev.codfw.wmnet'

=== 2024-04-17 ===
* 14:12 dcaro: deleting dns leaks
* 01:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])

=== 2024-04-16 ===
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 20:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2001-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2001-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T356287|T356287]])

=== 2024-04-15 ===
* 11:19 taavi: update spicerack to 8.5.0 on cloudcumin2001

=== 2024-04-10 ===
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-09 ===
* 18:13 andrewbogott: rebooting cloudinfra-cloudvps-puppetserver-1; unresponsive
* 13:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-05 ===
* 14:37 taavi: run maintain-replica-indexes on all web replicas [[phab:T361945|T361945]]
* 14:27 taavi: run maintain-replica-indexes on remaining analytics replicas [[phab:T361945|T361945]]
* 14:17 taavi: run maintain-replica-indexes on clouddb1017 [[phab:T361945|T361945]]

=== 2024-04-04 ===
* 18:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-03 ===
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T319184|T319184]])
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T319184|T319184]])
* 10:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T319184|T319184]])
* 10:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T319184|T319184]])
* 09:17 taavi: manually delete prometheus-node-textfile-wmcs-dnsleaks.service and related files from cloudservices1005/6, leftovers of the designate api to cloudcontrol migration
* 09:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T319184|T319184]])
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2024-04-02 ===
* 15:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T319184|T319184]])
* 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T319184|T319184]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 12:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 12:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-03-27 ===
* 11:47 taavi: deleting about 2k stale puppet certs by running wmcs-puppetcertleaks in delete mode

=== 2024-03-22 ===
* 10:25 dcaro: adding back cloudcephosd1034 to the pool after doing the performance tests ([[phab:T348643|T348643]])

=== 2024-03-21 ===
* 22:07 andrewbogott: doing dist-upgrade on cloudcontrol nodes to get mariadb upgraded for [[phab:T357133|T357133]]
* 22:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T357133|T357133]])
* 22:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T357133|T357133]])
* 11:08 dcaro: restarting nova-api on cloudcontrol1007

=== 2024-03-20 ===
* 17:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:27 dcaro: turning off cloudcephosd1030 to swap some disks ([[phab:T348643|T348643]])
* 15:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T348643|T348643]])
* 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2024-03-19 ===
* 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 13:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2024-03-18 ===
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-03-09 ===
* 16:43 andrewbogott: restarted nova-api on cloudcontrol1006

=== 2024-03-08 ===
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:21 arturo: restarted nova-api in cloudcontrol1007, it was complaining about mysql broken pipe
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 10:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 10:35 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2024-03-07 ===
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance

=== 2024-03-06 ===
* 17:46 dhinus: running &quot;wmcs-dnsleaks --delete&quot; to clean up 2 leaked records (tools-sgeweblight-10-32)
* 15:36 dcaro: renewing puppet ca cert for cloud-puppetmaster-03
* 15:24 dcaro: renewing puppet ca cert for cloudinfra-internal puppetmaster

=== 2024-03-04 ===
* 14:56 dhinus: delete project &quot;loggerdiscordbot&quot; in favor of new project &quot;discordbots&quot; [[phab:T358337|T358337]],[[phab:T358427|T358427]]
* 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reboot_node (exit_code=0) ([[phab:T359049|T359049]])
* 12:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reboot_node ([[phab:T359049|T359049]])

=== 2024-03-01 ===
* 14:59 taavi: removing wmf-auto-restart-cron from all VMs without cron via cumin - https://gerrit.wikimedia.org/r/c/operations/puppet/+/1007328/ [[phab:T358343|T358343]]
* 12:07 dcaro: restarted nova-api on cloudcontrol100* as it was very slow
* 12:04 dcaro: restarted nova-api on cloudcontrol1005 as it was very slow

=== 2024-02-27 ===
* 18:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-26 ===
* 10:02 arturo: deleting nskaggs account from gerrit's wmcs-trusted group

=== 2024-02-22 ===
* 13:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-21 ===
* 13:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) ([[phab:T319184|T319184]])
* 13:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance ([[phab:T319184|T319184]])
* 13:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T319184|T319184]])
* 13:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T319184|T319184]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-02-20 ===
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:45 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=0) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])

=== 2024-02-19 ===
* 12:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 10:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=0) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=99) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])

=== 2024-02-15 ===
* 17:34 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.roll_reboot_cloudnets ([[phab:T356975|T356975]])
* 15:34 taavi: restart radosgw in eqiad as I am seeing 500 errors
* 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:58 dhinus: restore correct aggregate &quot;localdisk&quot; for cloudvirtlocal1001 and remove &quot;maintenance&quot;
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T319184|T319184]])
* 05:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet[B&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet[B&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2024-02-14 ===
* 22:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtXXXX.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtXXXX.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:49 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:49 wmbot~andrew@bullseye: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97)
* 19:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:48 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:45 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:45 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:38 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:38 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:36 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:36 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:34 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:34 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:32 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 17:13 wmbot~fran@wmf3169: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtlocal1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T356975|T356975]])
* 17:12 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtlocal1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T356975|T356975]])
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:09 taavi: creating some missing $PROJECT.wmcloud.org. DNS zones
* 14:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:21 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 09:31 taavi: failover all dumps traffic to clouddumps1001
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 08:16 taavi: reboot clouddumps1001 for kernel updates

=== 2024-02-13 ===
* 17:07 wmbot~taavi@runko: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:07 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:06 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:06 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::eqiad1::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::eqiad1::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P:openstack::eqiad1::nova::compute::service'
* 16:02 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P:openstack::eqiad1::nova::compute::service'
* 16:01 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:01 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:10 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) ([[phab:T356975|T356975]])
* 14:55 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T356975|T356975]])

=== 2024-02-12 ===
* 17:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0)
* 17:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0)
* 17:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:23 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99)
* 17:21 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 17:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 17:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:36 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 16:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 15:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 15:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99)
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 01:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:34 andrewbogott: resetting eqiad1 rabbitmq in hopes of resolving neutron double message warnings
* 01:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-11 ===
* 21:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:57 andrewbogott: running wmcs.openstack.restart_openstack for all eqiad1 services
* 20:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 20:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-08 ===
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:43 taavi: deploy change to exclude cloud-private networks from general egress NAT https://phabricator.wikimedia.org/T356850

=== 2024-02-06 ===
* 20:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:09 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 02:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-05 ===
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:55 andrewbogott: rebuilt bookworm base image in eqiad1 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/992677

=== 2024-02-02 ===
* 13:54 arturo: [codfw1dev] cleanup /etc/network/interfaces on cloudlb2003-dev from puppet leftovers

=== 2024-02-01 ===
* 10:55 taavi: invite aborrero to /repos/cloud, /toolforge-repos, /cloudvps-repos on gitlab

=== 2024-01-29 ===
* 12:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-26 ===
* 09:01 taavi: joining cloudrabbit1001/2 to the cluster on 1003 [[phab:T345610|T345610]]

=== 2024-01-25 ===
* 16:48 andrewbogott: taavi just moved all rabbitmq traffic to cloudrabbit1003 as part of [[phab:T345610|T345610]]
* 16:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:27 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:27 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-2.tools.eqiad1.wikimedia.cloud to the cluster
* 13:15 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:15 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-1.tools.eqiad1.wikimedia.cloud to the cluster
* 12:48 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 12:48 wmbot~taavi@runko: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-1.toolsbeta.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-24 ===
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-22 ===
* 16:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:56 andrewbogott: restarting openstack sevices on eqiad1 to clean up from the mariadb restarts
* 15:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:07 andrewbogott: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/992192 and resetting galera cluster in eqiad1

=== 2024-01-21 ===
* 05:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 05:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-19 ===
* 23:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-18 ===
* 20:10 taavi: mysql:labsdbaccounts@m5-master.eqiad.wmnet [labsdbaccounts]&gt; update account_host set status = 'absent' where id = 137613;
* 12:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 12:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-101.tools.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-17 ===
* 15:17 andrewbogott: &quot;systemctl restart mariadb@s4.service mariadb@s6.service&quot; on clouddb1015. System is in danger of oom and there are no obvious long queries running

=== 2024-01-16 ===
* 14:25 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:25 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 13:55 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:55 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:54 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:54 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:53 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 13:53 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:52 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:52 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:35 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 11:34 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 11:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 09:41 taavi: drop dbproxy1018/9 grants from all clouddb hosts [[phab:T346947|T346947]]
* 09:24 taavi: move cloudvirt2004-dev from 'failed' to 'active' in netbox - seems like that was for [[phab:T348531|T348531]] which is now resolved

=== 2024-01-15 ===
* 14:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 14:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 14:32 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 12:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-11 ===
* 10:51 dcaro: restarting striker.service on cloudweb1003 as it seems non-responsive

=== 2024-01-10 ===
* 17:43 bd808: Blocking Developer accounts connected to invalid/legacy wikimedia.org email addresses ([[phab:T218239|T218239]])
* 16:48 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0) ([[phab:T346631|T346631]])
* 16:48 wmbot~fran@wmf3169: test message3 from local cookbook ([[phab:T346631|T346631]])
* 16:48 wmbot~fran@wmf3169: START - Cookbook wmcs.do_log_msg ([[phab:T346631|T346631]])

=== 2024-01-09 ===
* 17:50 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0) ([[phab:T346631|T346631]])
* 17:49 wmbot~fran@wmf3169: test message2 from local cookbook ([[phab:T346631|T346631]])
* 17:49 wmbot~fran@wmf3169: START - Cookbook wmcs.do_log_msg ([[phab:T346631|T346631]])
* 17:43 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])
* 17:43 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])
* 17:42 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])

=== 2024-01-08 ===
* 15:52 taavi: verify wmcloud.org, wmflabs.org and toolforge.org in gmail postmaster console to figure out how much google likes us ([[phab:T354112|T354112]])

=== 2024-01-07 ===
* 19:34 andrewbogott: removed cloudvirt1063 from 'ceph' aggregate, added to 'maintenance' aggregate   [[phab:T353408|T353408]]
* 19:34 andrewbogott: evacuating all VMs from cloudvirt1063.  [[phab:T353408|T353408]]

=== 2024-01-02 ===
* 16:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T353408|T353408]])
* 16:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T353408|T353408]])
* 10:22 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:22 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-12-31 ===
* 21:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:35 andrewbogott: running openstack service restart cookbook in eqiad1 in response to a bunch of service down alerts
* 21:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-21 ===
* 16:52 dhinus: puppet node deactivate cloudvirt1063.eqiad.wmnet [[phab:T353406|T353406]]
* 03:01 andrewbogott: restarting mariadb on cloudcontrol1005, hoping to get Galera back in sync

=== 2023-12-20 ===
* 19:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-18 ===
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:23 andrewbogott: restarting all eqiad1 openstack services after a rabbitmq upgrade/rebuild for [[phab:T353646|T353646]]
* 15:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-15 ===
* 13:13 dcaro: restarted nova-fullstack on codfw as it was stuck (and alerting through stale prometheus file)

=== 2023-12-14 ===
* 00:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 00:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 00:25 andrewbogott: evacuating hosts from cloudvirt1063 and depooling. [[phab:T353406|T353406]]

=== 2023-12-13 ===
* 16:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=99)

=== 2023-12-12 ===
* 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 17:45 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-100.tools.eqiad1.wikimedia.cloud to the cluster
* 16:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:11 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-99.tools.eqiad1.wikimedia.cloud to the cluster
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:49 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-98.tools.eqiad1.wikimedia.cloud to the cluster

=== 2023-12-10 ===
* 18:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:27 andrewbogott: restarting all openstack API servers, hoping to make things a bit more responsive
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-08 ===
* 12:00 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99)  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud ([[phab:T353055|T353055]])
* 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud ([[phab:T353055|T353055]])
* 11:58 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=97)  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud
* 11:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud
* 09:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) ([[phab:T345084|T345084]])
* 09:32 dcaro: restarting nova and keystone as they are getting too slow ([[phab:T345084|T345084]])
* 09:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack ([[phab:T345084|T345084]])
* 09:32 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) ([[phab:T345084|T345084]])
* 09:31 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack ([[phab:T345084|T345084]])

=== 2023-12-07 ===
* 13:12 dcaro: rebooting cloudcephosd1001 to make sure puppet7 migration went ok

=== 2023-12-04 ===
* 00:08 andrewbogott: rebooting cloudcontrol1006 to recover from full disk error

=== 2023-12-03 ===
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-02 ===
* 12:27 taavi: powercycle cloudvirt1063 [[phab:T352595|T352595]]
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:28 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-97.tools.eqiad1.wikimedia.cloud to the cluster
* 11:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-96.tools.eqiad1.wikimedia.cloud to the cluster
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 10:50 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-95.tools.eqiad1.wikimedia.cloud to the cluster
* 00:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:21 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-94.tools.eqiad1.wikimedia.cloud to the cluster
* 00:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-93.tools.eqiad1.wikimedia.cloud to the cluster
* 00:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-92.tools.eqiad1.wikimedia.cloud to the cluster
* 00:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-91.tools.eqiad1.wikimedia.cloud to the cluster
* 00:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:05 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-90.tools.eqiad1.wikimedia.cloud to the cluster
* 00:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:01 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-89.tools.eqiad1.wikimedia.cloud to the cluster

=== 2023-12-01 ===
* 17:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:19 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T351171|T351171]])
* 16:19 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 15:49 andrewbogott: reimaging cloudcontrol1005 due to widespread misbehavior
* 14:24 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T351171|T351171]])
* 14:20 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 14:19 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97) ([[phab:T351171|T351171]])
* 14:18 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 13:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:55 taavi: restart neutron-rpc-server.service on eqiad1 cloudcontrols

=== 2023-11-30 ===
* 20:44 andrewbogott: generating to application credentials for the tests that run on tf-infra-test
* 19:54 andrewbogott: reimaged cloudrabbit100[23] after https://gerrit.wikimedia.org/r/c/operations/puppet/+/979127.  I didn't reimage 1001 because that will require rebuilding the whole cluster but I did remove the related packages.
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet'
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet'
* 17:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet'
* 17:47 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet'
* 17:42 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:30 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:19 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:19 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:09 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:59 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:54 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:39 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:34 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:15 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:10 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:05 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:00 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:31 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:22 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:17 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:17 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:13 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:07 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:38 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:16 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:03 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 13:55 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 13:43 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T348843|T348843]])
* 13:43 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T348843|T348843]])
* 12:18 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 12:04 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:59 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:45 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:44 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:28 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T348843|T348843]])

=== 2023-11-29 ===
* 15:27 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:15 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:59 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T348843|T348843]])
* 14:50 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T348843|T348843]])

=== 2023-11-28 ===
* 14:18 taavi: moving wiki replica DNS to use cloudlbs instead of the old proxy VMs [[phab:T346947|T346947]]

=== 2023-11-27 ===
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-24 ===
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-22 ===
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 13:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-21 ===
* 10:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-20 ===
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-17 ===
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-16 ===
* 12:09 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:05 taavi@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 11:23 dhinus: upgraded spicerack from 8.0.2 to 8.0.3 on cloudcumins
* 05:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 05:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-15 ===
* 09:50 taavi: move cloudlb hosts to use the nftables firewall backend

=== 2023-11-14 ===
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 12:03 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T345811|T345811]])
* 11:44 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T345811|T345811]])
* 10:10 taavi: restart kiwix-mirror-update on clouddumps1001
* 05:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 05:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 04:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 04:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 04:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 04:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 03:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 03:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 03:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 03:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])

=== 2023-11-13 ===
* 22:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 22:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 22:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 22:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 21:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 19:31 andrewbogott: rebooting cloudcontrol2005-dev, trying to fix general misbehavior
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 19:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1033.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1032.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet'
* 17:09 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T345811|T345811]])
* 17:09 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T345811|T345811]])
* 16:57 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T345811|T345811]])
* 15:37 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T345811|T345811]])
* 15:19 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T345811|T345811]])
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-11 ===
* 02:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 00:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1058'
* 00:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1058'
* 00:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 00:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot

=== 2023-11-09 ===
* 21:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:54 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1025.eqiad.wmnet' ([[phab:T345811|T345811]])
* 14:52 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1025.eqiad.wmnet' ([[phab:T345811|T345811]])
* 14:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 14:50 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])

=== 2023-11-08 ===
* 16:43 andrewbogott: created foundationmemory project for [[phab:T350760|T350760]]

=== 2023-11-06 ===
* 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 15:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)

=== 2023-11-05 ===
* 00:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-11-04 ===
* 23:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 22:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 04:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)

=== 2023-11-03 ===
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:13 dhinus: triggering neutron failover from cloudnet1005 to cloudnet1006 ([[phab:T345811|T345811]])
* 06:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 01:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 01:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)

=== 2023-11-02 ===
* 20:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:32 taavi: merged cloudcontrol firewall cleanup patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/971211
* 16:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 16:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 13:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 11:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 11:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 07:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 07:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 05:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 03:47 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 03:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)

=== 2023-11-01 ===
* 23:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 21:29 taavi: re-enable puppet on cloudcontrol2006-dev.codfw.wmnet which has fallen off of puppetdb
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 20:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 19:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 14:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 09:04 taavi: reset local cookbook changes on cloudcumin1001 which were causing issues with puppet runs
* 09:02 taavi: restart nova-fullstack which had had some issues after yesterday's cloudcontrol1007 reimage
* 02:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 00:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-10-31 ===
* 23:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 18:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 14:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 10:16 dhinus: upgrading mariadb-server in cloudcontrol1005 ([[phab:T345811|T345811]])
* 10:04 dhinus: upgrading mariadb-server in cloudcontrol1006 ([[phab:T345811|T345811]])
* 09:51 dhinus: upgrading mariadb-server in cloudcontrol1007, second attempt ([[phab:T345811|T345811]])
* 06:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 05:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 02:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 02:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 02:27 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 02:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-10-30 ===
* 17:40 andrewbogott: rebooting tools-db-1.tools.eqiad1.wikimedia.cloud for yet another oom death
* 17:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 16:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 dhinus: upgrading mariadb-server in cloudcontrol1007 ([[phab:T345811|T345811]])
* 16:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-28 ===
* 07:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)

=== 2023-10-27 ===
* 15:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 15:38 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 15:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 15:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 15:29 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 15:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 15:27 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:09 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:05 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 09:04 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-26 ===
* 14:02 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 09:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-25 ===
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
* 11:00 taavi: update cloudcumins to spicerack 8.x
* 10:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster

=== 2023-10-24 ===
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-10-23 ===
* 16:48 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:06 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project catalyst in eqiad1
* 15:06 wm-bot2: fran@wmf3169 START - Cookbook wmcs.vps.create_project for project catalyst in eqiad1
* 10:36 taavi: merged change https://gerrit.wikimedia.org/r/c/operations/puppet/+/966494 which touches the pdns web server config

=== 2023-10-20 ===
* 15:17 dcaro: upgraded cloudcephosd1004 to v15 ([[phab:T349363|T349363]])
* 14:25 dcaro: upgraded cloudcephosd1003 to v15 ([[phab:T349363|T349363]])
* 13:20 dcaro: upgraded cloudcephosd1002 to v15 ([[phab:T349363|T349363]])
* 10:33 dcaro: upgraded cloudcephosd1001 to v15 ([[phab:T349363|T349363]])
* 08:26 dcaro: codfw ceph enabled diskprediction_local module, will take a bit to populate/start getting predictions ([[phab:T348716|T348716]])

=== 2023-10-17 ===
* 15:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T349109|T349109]])
* 15:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T349109|T349109]])

=== 2023-10-16 ===
* 03:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-10-13 ===
* 23:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 08:31 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 08:30 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 08:20 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])

=== 2023-10-12 ===
* 17:16 wm-bot2: fran@wmf3169 END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) ([[phab:T341285|T341285]])
* 17:16 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])

=== 2023-10-11 ===
* 10:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 10:36 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 07:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 06:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-10 ===
* 17:25 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:50 dcaro: removing ~100 dangling backup snapshots from eqiad1-compute ceph pool
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 12:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 11:38 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 11:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 11:33 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 11:32 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 11:00 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) ([[phab:T341285|T341285]])
* 11:00 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 10:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 09:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 09:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 09:43 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 08:19 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:18 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:18 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:17 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:17 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-09 ===
* 17:16 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:26 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:18 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:49 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:26 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 13:55 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 13:45 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 13:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:04 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:10 dcaro: undrained cephosd1011
* 09:09 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 09:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:06 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 09:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:05 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 09:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 07:35 taavi: restart postgresql on cloudbackup2001 [[phab:T348431|T348431]]

=== 2023-10-05 ===
* 16:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:29 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 16:24 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:11 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:30 arturo: operating on cloudgw @ eqiad1 ([[phab:T347469|T347469]])
* 14:07 wm-bot2: dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 12:55 arturo: doing cloudgw maintenance operations [[phab:T347469|T347469]]
* 11:54 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_rack
* 10:57 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 09:54 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 09:52 arturo: [codfw1dev] aborrero@cloudcontrol2001-dev:~ $ sudo keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
* 09:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 09:40 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 09:33 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 08:50 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 00:04 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)

=== 2023-10-04 ===
* 20:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 15:55 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 14:54 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 14:44 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 14:41 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 14:40 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 13:50 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:08 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:21 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:12 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-03 ===
* 19:38 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:35 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:32 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 11:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 08:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:43 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 08:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:39 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 08:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:23 dcaro: set .rgw.root pool on eqiad as rgw app (`ceph osd pool application enable .rgw.root rgw`)

=== 2023-10-02 ===
* 17:39 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:30 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:58 taavi: cloudcontrol1005,7: `sudo systemctl reset-failed keystone_sync_keys_from_cloudcontrol1006.eqiad.wmnet.service`
* 13:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:40 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:37 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 12:26 arturo: [codfw1dev] run `update domains set master = '185.15.57.25:5354 185.15.57.26:5354 172.20.5.9:5354 172.20.5.8:5354';` in cloudservies2005-dev
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])
* 11:55 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])
* 11:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 08:44 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])

=== 2023-09-29 ===
* 12:43 taavi: taavi@cloudcontrol1005 ~ $ os subnet set a69bdfad-d7d2-4cfa-8231-{{Gerrit|3d6d3e0074c9}} --no-dns-nameservers --dns-nameserver 172.20.255.1
* 08:36 taavi: start script to fix networking on broken bullseye instances [[phab:T347665|T347665]]

=== 2023-09-28 ===
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 12:01 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:48 arturo: rebooting cloudgw1001/1002 for sysctl and kernel upgrades

=== 2023-09-27 ===
* 12:07 arturo: merging cloudgw firewall changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/961360
* 09:36 taavi: move maintain-dbusers to cloudcontrol1005
* 01:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-26 ===
* 17:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-24 ===
* 15:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-23 ===
* 08:40 taavi: restart keystone

=== 2023-09-22 ===
* 14:03 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 14:02 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 14:02 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 14:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 14:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.undrain_node (exit_code=0)
* 14:00 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 14:00 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:56 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.undrain_node (exit_code=0)
* 13:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.undrain_node (exit_code=99)
* 13:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:48 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.undrain_node (exit_code=99)
* 13:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 13:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:46 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:44 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:41 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:37 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:33 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:33 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 11:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node

=== 2023-09-21 ===
* 16:35 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0)  on tools-db-3.tools.eqiad1.wikimedia.cloud
* 16:34 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs  on tools-db-3.tools.eqiad1.wikimedia.cloud
* 02:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-20 ===
* 21:49 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:45 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:38 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 21:35 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:31 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:12 arturo: moving openstack API endpoint to cloudlb ([[phab:T346439|T346439]])
* 10:28 arturo: running SQL command `update domains set master=&quot;172.20.1.5:5354 172.20.2.4:5354 185.15.56.162:5354 185.15.56.163:5354&quot;;` on cloudservices1005/1006 ([[phab:T346042|T346042]])
* 10:06 arturo: running SQL command `update domains set master=&quot;185.15.56.162:5354 185.15.56.163:5354&quot;` on cloudservices1005/1005 ([[phab:T346042|T346042]])

=== 2023-09-19 ===
* 18:54 andrewbogott: depooling clouddb1019 to let it recover from high memory use.  [[phab:T346826|T346826]]
* 15:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-18 ===
* 11:53 taavi: update designate urls in Keystone to point to openstack-next, until cloudlb is serving the main openstack address [[phab:T346042|T346042]]
* 11:40 arturo: decomission cloudservices1005 [[phab:T346042|T346042]] in preparation for re-racking
* 08:45 arturo: hardcode `185.15.56.161 openstack.eqiad1.wikimediacloud.org` in /etc/hosts in cloudcontrol1005 for [[phab:T346441|T346441]]

=== 2023-09-15 ===
* 11:43 arturo: merging NAT change for [[phab:T346426|T346426]] in cloudgw
* 10:33 arturo: faiolver cloudgw1001 into cloudgw1002, investigating a nftables syntax error ([[phab:T346432|T346432]])

=== 2023-09-14 ===
* 21:25 wm-bot2: andrew@bullseye END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:25 wm-bot2: andrew@bullseye START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:23 wm-bot2: andrew@bullseye END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:23 wm-bot2: andrew@bullseye START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 17:13 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 17:06 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 17:01 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:51 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:29 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:29 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:25 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) ([[phab:T345810|T345810]])
* 16:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:12 topranks: DNS operation: remove old DNS entry for ns0.openstack.eqiad1.wikimediacloud.org. on wikimedia authdns (was pointing to 208.80.154.148)
* 16:10 topranks: DNS operation: add new DNS entry for ns0.openstack.eqiad1.wikimediacloud.org. on wikimedia authdns pointing to 185.15.56.162
* 14:42 arturo: DNS operation: route 208.80.154.148 to cloudservices1006 in anticipation of cloudservices1005 decom ([[phab:T346042|T346042]])
* 12:11 arturo: enable puppet on cloudservices1006 to drop local NAT hacks and enable new DNS auth IP address ([[phab:T346042|T346042]])

=== 2023-09-13 ===
* 17:11 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) ([[phab:T345811|T345811]])
* 17:08 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 17:04 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 17:04 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:57 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:53 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:53 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:49 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:49 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 01:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 01:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 01:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-12 ===
* 18:06 andrewbogott: update domains set master='185.15.56.163:5354 208.80.154.11:5354 10.64.151.4:5354'; on cloudservices1005 + cloudservices1006
* 17:59 andrewbogott: mysql:root@localhost [pdns]&gt; update domains set master='185.15.56.163:5354 208.80.154.11:5354';
* 17:59 andrewbogott: &quot;designate-manage pool update' on cloudservices1005 to remove cloudservices1004 from the pool
* 17:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-11 ===
* 12:36 arturo: update DNS resolver cloud-wide to use 172.20.255.1 ([[phab:T342621|T342621]])

=== 2023-09-10 ===
* 02:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 02:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-07 ===
* 10:09 dhinus: reimaging cloudcontrol2001-dev to bookworm ([[phab:T345810|T345810]])

=== 2023-09-06 ===
* 16:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97)
* 16:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:53 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:37 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:37 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:37 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:35 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:35 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:34 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:34 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:23 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97)
* 16:23 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:13 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99)
* 16:12 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance

=== 2023-09-05 ===
* 12:34 arturo: synced pdns database from cloudservices1004 to cloudservices1006 ([[phab:T345240|T345240]])
* 12:18 arturo: updating pools.yaml in all cloudservices designate nodes ([[phab:T345240|T345240]])
* 10:54 arturo: running SQL command `update domains set master=&quot;208.80.154.11:5354 208.80.154.148:5354 10.64.151.4:5354&quot;;` on all 3 cloudservices nodes ([[phab:T345240|T345240]])

=== 2023-09-04 ===
* 15:58 arturo: stop and mask designate-sink.service @ cloudservices1006
* 14:19 arturo: started all designate services on cloudservices1006 [[phab:T345240|T345240]]
* 10:46 arturo: added designate galera DB grants for cloudlb [[phab:T345240|T345240]]
* 08:40 arturo: stopped all designate services on cloudservices1006 [[phab:T345240|T345240]]

=== 2023-09-01 ===
* 10:28 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.network.tests (exit_code=1) ([[phab:T345282|T345282]])
* 10:25 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.network.tests ([[phab:T345282|T345282]])

=== 2023-08-31 ===
* 15:14 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=99)
* 15:14 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 12:49 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudnet.show (exit_code=0)
* 12:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudnet.show
* 12:48 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 12:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 12:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 12:46 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:46 wm-bot2: fran@wmf3169 START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2023-08-30 ===
* 13:07 wm-bot2: dcaro testing stuff

=== 2023-08-28 ===
* 15:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:05 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-23 ===
* 16:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:10 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:08 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:08 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-15 ===
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:32 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 19:32 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:01 andrewbogott: rebooting cloudvirt2001-dev in an attempt to figure out what's happening with bastions
* 15:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:42 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:39 dcaro: removed some logs from the cloudmetrics1003:/var/log/carbon/ directory and stopped the carbon processes (they were crashing and filling up the disk with logs)

=== 2023-08-14 ===
* 22:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:11 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:08 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-07 ===
* 09:39 taavi: cloud vps graphite service was disabled: https://wikitech.wikimedia.org/wiki/News/2023_Cloud_VPS_metrics_changes [[phab:T326266|T326266]]

=== 2023-08-03 ===
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0)
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.do_log_msg
* 13:07 wm-bot2: Test SAL log ([[phab:T341793|T341793]]) - cookbook ran by root@cloudcumin1001

=== 2023-07-31 ===
* 16:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0)
* 16:16 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 16:15 fnegri@cloudcumin1001: START - Cookbook wmcs.do_log_msg
* 15:33 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:42 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:35 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 13:55 andrewbogott: recreating the codfw1dev galera cluster according to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Galera -- mariadb is stopped (and won't start) on all three cloudcontrol nodes
* 13:39 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001

=== 2023-07-27 ===
* 10:52 arturo: adding cloud-private subnet to cloudnet1005/1006 hosts in eqiad1 ([[phab:T342619|T342619]])

=== 2023-07-19 ===
* 14:02 wm-bot2: Draining cloudvirt2002-dev.codfw.wmnet ([[phab:T335840|T335840]]) - cookbook ran by raymond@ubuntu
* 14:02 wm-bot2: Safe rebooting cloudvirt2002-dev.codfw.wmnet ([[phab:T335840|T335840]]) - cookbook ran by raymond@ubuntu

=== 2023-07-17 ===
* 15:55 arturo: cloudcontrol1005 was shutdown earlier today ([[phab:T341495|T341495]])
* 15:35 arturo: [codfw1dev] cloudweb2002-dev up and running after reracking ([[phab:T327919|T327919]])
* 15:11 arturo: [codfw1dev] powered off cloudweb2002-dev for reracking ([[phab:T327919|T327919]])
* 12:45 taavi: removing diamond from remaining buster instances [[phab:T317032|T317032]]

=== 2023-07-13 ===
* 10:48 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:38 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:33 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay

=== 2023-07-07 ===
* 10:28 taavi: backfilling &lt;nowiki&gt;{&lt;/nowiki&gt;project&lt;nowiki&gt;}&lt;/nowiki&gt;.wmcloud.org and other currently-named DNS zones to projects that don't have them

=== 2023-07-06 ===
* 17:02 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:00 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:00 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-07-04 ===
* 14:08 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:07 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001

=== 2023-06-30 ===
* 21:37 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-06-29 ===
* 19:49 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:47 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-06-26 ===
* 08:46 arturo: [codfw1dev] manually start mariadb service @ cloudinfra-db-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud

=== 2023-06-24 ===
* 19:21 wm-bot2: Created new flavor: g3.cores16.ram34.disk20 (id:7dd33202-32c3-4bc7-b2d4-{{Gerrit|10c2ebe7e5c5}}) - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Created new flavor: g3.cores16.ram34816.disk20.admin (id:d76925a8-b58b-489e-a3d9-{{Gerrit|c68c7744b551}}) - cookbook ran by andrew@bullseye

=== 2023-06-23 ===
* 16:45 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 13:59 andrewbogott: rebooting every VM in codfw1dev

=== 2023-06-13 ===
* 15:35 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 14:58 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye

=== 2023-06-12 ===
* 19:37 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 11:57 arturo: [codfw1dev] refresh various occurrences of old FQDNs in instance puppet via horizon ([[phab:T324992|T324992]])

=== 2023-06-08 ===
* 16:56 andrewbogott: deleting all Stretch base images from glance
* 16:45 andrewbogott: updated the bullseye image with https://cloud.debian.org/images/cloud/bullseye/20230601-1398/debian-11-genericcloud-amd64-20230601-1398.tar.xz
* 12:17 wm-bot2: Drained cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:17 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 02920314-1efe-4934-ad81-{{Gerrit|d2a6cf2e17ab}}, use this to unset) ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:16 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:06 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 769349bf-465f-4f0c-a8f3-{{Gerrit|f2423631ba7e}}, use this to unset) ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus

=== 2023-06-07 ===
* 10:06 dcaro: upgraded ruby2.5 to latest fixed version on all buster VMs
* 08:10 dcaro: downgrading ruby2.5 to previous backport on all buster VMs

=== 2023-06-06 ===
* 19:09 andrewbogott: also increased RAM and secgroup-rule quota for Trove  [[phab:T337882|T337882]]
* 19:06 andrewbogott: increased trove secgroups, instances, volumes quotas from 40 to 100.  Trove is too popular!  [[phab:T337882|T337882]]
* 18:31 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 18:31 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:26 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:26 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:42 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:42 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye

=== 2023-06-05 ===
* 09:41 arturo: [codfw1dev] rebooting bastion-codfw1dev-02 (no IP address in the main interface) [[phab:T336963|T336963]]

=== 2023-06-02 ===
* 14:40 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by arturo@nostromo
* 12:59 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 12:58 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:58 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:44 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 12:44 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:43 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-26 ===
* 16:03 andrewbogott: &quot;maintain-views --all-databases --replace-all&quot; on clouddb1021 for [[phab:T337446|T337446]]

=== 2023-05-22 ===
* 19:54 andrewbogott: deleting project 'citelearn' as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge#SHUTDOWN_citelearn

=== 2023-05-21 ===
* 23:29 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 23:27 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:44 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:44 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-19 ===
* 14:53 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-18 ===
* 21:39 andrewbogott: deleting obsolete roles '4d8cad783d6342efa8414d7d36fbc034 {{!}} projectadmin_renamed_for_[[phab:T330759|T330759]]' and 'f473273fac7146b3bdbf22e5d4504f95 {{!}} user_renamed_for_[[phab:T330759|T330759]]' on eqiad1. State pre-deletion is dumped to /root/allassignmentspredeletion.txt  on cloudcontrol1007.
* 21:35 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:33 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:18 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:10 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Restarting openstack services on cloudcontrol2001-dev@local1: ['cinder-volume'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:20 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-15 ===
* 14:28 wm-bot2: Drained cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:28 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: 96ce2ed0-3aff-4d04-be0b-{{Gerrit|e16513070617}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:27 wm-bot2: Draining cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:23 wm-bot2: Drained cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 110176f8-04d5-4110-bb7d-{{Gerrit|1ab272bd8be2}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: c6e92e13-49f4-4db3-8a13-{{Gerrit|8692ccfd3bc9}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:12 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:12 wm-bot2: Drained cloudvirt1035.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:11 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: fa730dec-848f-45fb-9eda-{{Gerrit|e74bd874c5c9}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:10 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1006: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1007: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1005: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:01 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: eb1cfac0-d481-4baa-b9cd-{{Gerrit|15e5fbcef495}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:00 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:58 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: 3e6c3ff3-7d55-4777-9032-{{Gerrit|b867a257eced}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:57 wm-bot2: Draining cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:53 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: 0693664a-df78-417e-ba34-{{Gerrit|590e5a0a9981}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:52 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:49 wm-bot2: Set cloudvirt cloudvirt1035.eqiad.wmnet maintenance (downtime id: e6929ab8-4bc3-4186-817b-{{Gerrit|9b53dbd597c6}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:48 wm-bot2: Draining cloudvirt1035.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 andrewbogott: restarting nova services in eqiad1, trying to free up db connections
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:33 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: fef43d73-6fd4-4dde-a0ac-{{Gerrit|95fd69a9b0c1}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:32 wm-bot2: Draining cloudvirt1034.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 5f867662-e824-498c-a715-{{Gerrit|2e2ad50f0bb5}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: 0f515150-1313-41d4-a5f6-{{Gerrit|9bc00ce9b245}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Draining cloudvirt1033.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Drained cloudvirt1032.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot2: Set cloudvirt cloudvirt1032.eqiad.wmnet maintenance (downtime id: e826adc3-addd-44d8-b39e-{{Gerrit|ae7bd2df1e60}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:49 wm-bot2: Draining cloudvirt1032.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:49 wm-bot2: Drained cloudvirt1031.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:14 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 4154c818-744c-4d84-9883-{{Gerrit|cae7a5826ed5}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:13 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:13 wm-bot2: Drained cloudvirt1026.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:04 wm-bot2: Set cloudvirt cloudvirt1026.eqiad.wmnet maintenance (downtime id: cdfc3d01-ec1e-483c-9a38-{{Gerrit|834193e487ff}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Draining cloudvirt1026.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Drained cloudvirt1025.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 11:51 wm-bot2: Set cloudvirt cloudvirt1025.eqiad.wmnet maintenance (downtime id: 6a56757d-35de-499e-8209-{{Gerrit|728bcf62a22a}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 11:50 wm-bot2: Draining cloudvirt1025.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-12 ===
* 17:52 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-10 ===
* 16:09 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 andrewbogott: running &quot;cookbook -c ~/.config/spicerack/cookbook_config.yaml wmcs.openstack.restart_openstack --cluster-name eqiad1 --all&quot; to pick up changes for testing [[phab:T336379|T336379]]
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 03:05 andrewbogott: &quot;systemctl restart puppet-enc&quot; on enc-1.cloudinfra.eqiad1.wikimedia.cloud.  Seems to have crashed.

=== 2023-05-05 ===
* 16:07 wm-bot2: Drained cloudvirt1024.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:03 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 95995009-09d6-496e-8cd2-{{Gerrit|0cfac93d3cf7}}, use this to unset) ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:02 wm-bot2: Draining cloudvirt1024.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Drained cloudvirt1023.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:51 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 53c46cae-00af-4664-97ff-{{Gerrit|266b393335bb}}, use this to unset) ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:50 wm-bot2: Draining cloudvirt1023.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:49 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 528ea4f6-8088-475e-937f-{{Gerrit|098ffba861b6}}, use this to unset) - cookbook ran by andrew@bullseye
* 15:47 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 3ef85b5e-d9d9-4b24-901b-{{Gerrit|a3058a7d0615}}, use this to unset) - cookbook ran by andrew@bullseye
* 15:44 andrewbogott: moved cloudvirt1023 and cloudvirt1024 from 'ceph' aggregate to 'maintenance' aggregate, prep for decom [[phab:T336064|T336064]]
* 15:44 andrewbogott: moved cloudvirt1028 from 'localdisk' aggregate to 'maintenance' aggregate. Nothing new should be scheduled here, local storage should now move to cloudvirtlocal100x
* 15:41 andrewbogott: moved cloudvirt1055 and cloudvirt1056 from 'spare' to 'ceph' aggregate.  Prep for removing two obsolete cloudvirts, 1023 and 1024.  [[phab:T336064|T336064]]

=== 2023-05-04 ===
* 22:49 andrewbogott: removed fullstack-* puppet reports on puppetmaster-02.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud and cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs to free up disk space

=== 2023-05-02 ===
* 13:01 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:31 wm-bot2: Destroying OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 12:30 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Added 1 new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Added OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus

=== 2023-05-01 ===
* 17:09 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 17:09 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 17:08 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 17:08 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 15:22 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 13:53 taavi: running wmcs-novastats-puppetleaks in real mode [[phab:T334127|T334127]]
* 13:51 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus

=== 2023-04-18 ===
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1007: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1006: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1005: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudvirt1056: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1019: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1060: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1023: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1038: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1036: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1039: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1050: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1061: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1028: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1020: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1052: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1040: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1043: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1026: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1030: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1048: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1025: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1035: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1055: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1042: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1059: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1057: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1032: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1029: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1044: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1047: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1034: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1058: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1045: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1041: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1031: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1046: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1027: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1054: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1033: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1037: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1051: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1053: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1049: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1019: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1020: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:41 andrewbogott: resetting rabbitmq on cloudrabbit1003 due to splitbrain
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1019: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1020: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye

=== 2023-04-17 ===
* 08:49 wm-bot2: Increased quotas by 9 cores, 1 instances, 16 ram ([[phab:T334695|T334695]]) - cookbook ran by dcaro@vulcanus

=== 2023-04-06 ===
* 17:03 andrewbogott: running wmcs-wikireplica-dns on cloudcontrol1005 to update tools-db dns entries

=== 2023-04-04 ===
* 17:23 andrewbogott: resetting all three rabbitmq nodes and restarting all openstack services as per https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Rabbitmq#Resetting_the_HA_setup

=== 2023-03-28 ===
* 13:56 andrewbogott: depooling cloudweb1003 before switch upgrade
* 10:58 dhinus: disabled tool &quot;wb&quot; by clicking the disable button at https://toolsadmin.wikimedia.org/tools/id/wb [[phab:T328693|T328693]]
* 08:34 arturo: cleanup neutron agents for cloudvirt1021/1022 (decom)
* 08:32 arturo: cleanup neutron agents for cloudvirt1017 (decom)

=== 2023-03-27 ===
* 14:30 wm-bot2: Drained cloudvirt1024.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:19 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 3f43d3ca-696c-4d3c-8d5b-{{Gerrit|e57984f0eb86}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:18 wm-bot2: Draining cloudvirt1024.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:17 wm-bot2: Drained cloudvirt1023.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:08 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: f4767781-ea26-453b-9521-{{Gerrit|847a8340a249}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:07 wm-bot2: Draining cloudvirt1023.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:04 wm-bot2: Drained cloudvirt1022.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:55 wm-bot2: Set cloudvirt cloudvirt1022.eqiad.wmnet maintenance (downtime id: 0e794cfe-5896-46c1-842d-{{Gerrit|c34719140d4f}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:54 wm-bot2: Draining cloudvirt1022.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:54 wm-bot2: Drained cloudvirt1021.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:48 wm-bot2: Set cloudvirt cloudvirt1021.eqiad.wmnet maintenance (downtime id: e7a904ca-003d-450a-ad85-{{Gerrit|886fb80dfc41}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:47 wm-bot2: Draining cloudvirt1021.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:46 wm-bot2: Drained cloudvirt1017.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Set cloudvirt cloudvirt1017.eqiad.wmnet maintenance (downtime id: 0ff0090f-26c3-4278-a9c9-{{Gerrit|cce518559408}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:35 wm-bot2: Draining cloudvirt1017.eqiad.wmnet - cookbook ran by andrew@bullseye

=== 2023-03-22 ===
* 12:41 taavi: delete wmde-templates-alpha project [[phab:T332773|T332773]]

=== 2023-03-08 ===
* 21:45 bd808: maintain-kubeusers container in CrashLoopBackoff, investigating
* 13:49 dcaro: stopping puppet on labostre1004 to debug maintain-dbusers

=== 2023-03-07 ===
* 16:06 andrewbogott: updated application credential roles, replacing 'user' with 'reader' and 'projectadmin' with 'member':  update application_credential_role set role_id='f75a3c410bca4e96a1cf6ac103b0ccaf' where role_id='f473273fac7146b3bdbf22e5d4504f95'  and update application_credential_role set role_id='38676f30eaeb44518bf7e144a73c8da6' where role_id='4d8cad783d6342efa8414d7d36fbc034'
* 10:11 dcaro: there was a little unavailability for some VMs while ceph was starting to rebalance things, but it seems stable and moving data around ([[phab:T331141|T331141]])
* 09:37 dcaro: Changing ceph crush map to allow rack HA on eqiad1 cluster ([[phab:T331141|T331141]])

=== 2023-03-03 ===
* 12:12 arturo: installing haproxy updates ([[phab:T331119|T331119]])

=== 2023-03-02 ===
* 16:17 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1010.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Added 1 new OSDs ['cloudcephosd1010.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Added OSD cloudcephosd1010.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot2: Finished rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:10 wm-bot2: Rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Adding OSD cloudcephosd1010.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Adding new OSDs ['cloudcephosd1010.eqiad.wmnet'] to the cluster ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 10:43 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1005.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:44 wm-bot2: Added 1 new OSDs ['cloudcephosd1005.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:44 wm-bot2: Added OSD cloudcephosd1005.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:36 wm-bot2: Finished rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:33 wm-bot2: Rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Adding OSD cloudcephosd1005.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Adding new OSDs ['cloudcephosd1005.eqiad.wmnet'] to the cluster ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-28 ===
* 22:47 andrewbogott: adding new 'member' role assignment to every user/project pair that currently has the 'user' assignment.  [[phab:T330759|T330759]]
* 09:47 wm-bot2: Depooled and destroyed OSD daemons [79, 78, 77, 76, 75, 74, 73, 72] and removed the OSD host cloudcephosd1010 from the CRUSH map. ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:46 wm-bot2: Destroying OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:28 wm-bot2: Depooling OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot2: Depooling OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot2: Depooled and destroyed OSD daemons [39, 38, 37, 36, 35, 34, 33, 32] and removed the OSD host cloudcephosd1005 from the CRUSH map. ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:11 wm-bot2: Destroying OSDs with ids in [39, 38, 37, 36, 35, 34, 33, 32] on cloudcephosd1005 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:55 wm-bot2: Depooling OSDs with ids in [39, 38, 37, 36, 35, 34, 33, 32] on cloudcephosd1005 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-27 ===
* 21:01 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1004.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:53 wm-bot2: Added 1 new OSDs ['cloudcephosd1004.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:53 wm-bot2: Added OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:45 wm-bot2: Finished rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:42 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:41 wm-bot2: Adding OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:41 wm-bot2: Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Adding OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:14 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1003.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:13 wm-bot2: Added 1 new OSDs ['cloudcephosd1003.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:13 wm-bot2: Added OSD cloudcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:05 wm-bot2: Finished rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:02 wm-bot2: Rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding OSD cloudcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding new OSDs ['cloudcephosd1003.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:59 wm-bot2: Adding OSD coludcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:59 wm-bot2: Adding new OSDs ['coludcephosd1003.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:58 wm-bot2: Adding OSD coludcephosd1003... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:58 wm-bot2: Adding new OSDs ['coludcephosd1003'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-22 ===
* 08:25 wm-bot2: Depooled and destroyed OSD daemons [31, 30, 29, 28, 27, 26, 25, 24] and removed the OSD host cloudcephosd1004 from the CRUSH map. ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:22 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:15 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:13 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 07:23 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 07:02 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-21 ===
* 20:27 andrewbogott: deleted 200 more orphaned VM images with wmcs-novastats-cephleaks
* 17:18 andrewbogott: shutting down postgres on clouddb1004/1003, then shutting down the vms
* 15:50 wm-bot2: Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:48 wm-bot2: Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 14:00 wm-bot2: Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-16 ===
* 19:14 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1002.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:55 dcaro: Manually zapped /dev/sdc on cloudcephosd1002, probably a leftover drive since the beginning (or during the reimage the drives changed names, and this one had leftovers from the previous OS) ([[phab:T329498|T329498]])
* 17:47 wm-bot2: Added 1 new OSDs ['cloudcephosd1002.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:47 wm-bot2: Added OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:42 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:42 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:50 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:50 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:00 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:00 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot2: Added 1 new OSDs ['cloudcephosd1001.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Destroying OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot2: Depooling OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:20 wm-bot2: Destroying OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:19 wm-bot2: Depooling OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:03 wm-bot2: Destroying OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:01 wm-bot2: Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 10:59 wm-bot2: Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 10:15 dcaro: purges osd daemons 48 and 40 from eqiad ceph cluster ([[phab:T329709|T329709]])

=== 2023-02-15 ===
* 14:53 andrewbogott: deleting another 100 leaked VM images with wmcs-novastats-cephleaks
* 13:39 wm-bot2: Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:13 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:11 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:11 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot2: Destroying OSDs with id [[12345]] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:09 wm-bot2: Destroying OSDs with id ['12345'] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus

=== 2023-02-14 ===
* 13:17 andrewbogott: restarting all eqiad1 openstack services because that seems to sometimes help things *shrug*

=== 2023-02-13 ===
* 14:06 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 8fbf6bfd-eec1-4d81-8e0d-{{Gerrit|ea431d8411ee}} ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:32 taavi: re-enable puppet on labstore1004 [[phab:T329377|T329377]]

=== 2023-02-09 ===
* 21:17 andrewbogott: deleted 10% of leaked VM ceph images using wmcs-novastats-cephleaks (only 10% out of an abundance of caution)

=== 2023-02-08 ===
* 17:08 arturo: changing to cloudgw network setup, make VIPs /32 ([[phab:T295774|T295774]])

=== 2023-02-07 ===
* 11:26 arturo: [codfw1dev] testing network changes in cloudgw, expect unrealiable network ([[phab:T295774|T295774]])

=== 2023-02-04 ===
* 13:44 taavi: drop old columns from oathauth_users table on labtestwiki [[phab:T328131|T328131]]

=== 2023-02-03 ===
* 15:00 andrewbogott: restarted nova services in eqiad1 in an attempt to eke out another day or two of stability
* 14:13 taavi: attached GrapheSuppression developer account to wikitech

=== 2023-02-02 ===
* 13:14 dcaro_away: draining osd.48 from node cloudcephosd1001 ([[phab:T316544|T316544]])
* 12:57 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 7ac2b25a-d1bb-4789-8aa6-{{Gerrit|b9435b505349}} ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus

=== 2023-01-30 ===
* 22:34 wm-bot2: Upgraded and rebooted host cloudrabbit1002.wikimedia.org - cookbook ran by andrew@bullseye
* 21:34 andrewbogott: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/884922 and upgrading rabbitmq nodes for [[phab:T328155|T328155]]

=== 2023-01-27 ===
* 20:08 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:22 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 15:25 andrewbogott: restarting openstack services in eqiad1, another attempt to address instability

=== 2023-01-26 ===
* 20:34 andrewbogott: shutting down mariadb on cloudbackup2001-dev, testing the waters for [[phab:T328079|T328079]]

=== 2023-01-22 ===
* 03:42 andrewbogott: reset eqiad1 rabbitmq in an attempt to resolve some mild instability

=== 2023-01-20 ===
* 15:26 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 15:26 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['f47a3d91-b270-4c90-acc8-d85075a6bf8e'], use this to unset) - cookbook ran by andrew@bullseye
* 13:15 arturo: reinstall python3-neutron (to reset manual patching) on all cloudnet nodes and patch it via puppet, then restart neutron-l3-agent by hand ([[phab:T327463|T327463]])
* 10:12 arturo: [codfw1dev] failover neutron-l3-agent between cloudnet2005-dev/cloudnet2006-dev a couple of times [[phab:T327463|T327463]]
* 02:17 andrewbogott: stopping neutron-l3-agent on cloudnet1005 because it's logging at a furious rate and about to fill the drive

=== 2023-01-19 ===
* 18:06 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 18:06 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['36d5af6a-7d8e-4d0c-831e-1bf05c255984'], use this to unset) - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 17:22 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['66ec4f04-d25b-4067-be9e-2fe12cb1d3ff'], use this to unset) - cookbook ran by andrew@bullseye

=== 2023-01-18 ===
* 22:32 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 347cb75e-215e-4b85-ae14-{{Gerrit|4ce1934c70c7}}, use this to unset) - cookbook ran by andrew@bullseye
* 22:01 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 9bf6212f-7fdb-4869-8190-{{Gerrit|b07387b2bc7e}}, use this to unset) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 11aec8ea-f443-41ae-b79a-{{Gerrit|5e1e3aa94546}}, use this to unset) - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:03 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:38 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:38 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:37 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:37 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-01-17 ===
* 19:32 wm-bot2: Upgraded and rebooted host cloudbackup2002.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:04 wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye
* 17:52 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 17:23 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye

=== 2023-01-13 ===
* 17:36 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:26 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:26 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:25 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:21 wm-bot2: Restarting openstack services: &lt;nowiki&gt;{&lt;/nowiki&gt;'cloudcontrol2001-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'], 'cloudcontrol2004-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'], 'cloudvirt2001-dev': ['nova-compute'], 'cloudvirt2002-dev': ['nova-compute'], 'cloudvirt2003-dev': ['nova-compute'], 'cloudcontrol2005-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata
* 10:41 arturo: restart backup_vm.service on cloudbackup1003/1004 to recover from a nova Unknown Error (HTTP 503)

=== 2023-01-12 ===
* 22:34 andrewbogott: updated the Bullseye base image with the upstream {{Gerrit|20221219}} build
* 14:12 wm-bot2: Created project checkuser-beta-wiki with default quotas. ([[phab:T326740|T326740]]) - cookbook ran by arturo@nostromo

=== 2023-01-06 ===
* 18:42 wm-bot2: Safe reboot of cloudvirt2003-dev.codfw.wmnet finished successfully - cookbook ran by andrew@bullseye
* 18:42 wm-bot2: unset cloudvirt2003-dev.codfw.wmnet maintenance (aggregates: ceph) - cookbook ran by andrew@bullseye
* 18:39 wm-bot2: Drained cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: b50d9d3a-4f1d-4522-b86f-{{Gerrit|722fc9c55c87}}, use this to unset) - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Safe rebooting cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Draining cloudvirt2003-dev.cdofw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Safe rebooting cloudvirt2003-dev.cdofw.wmnet - cookbook ran by andrew@bullseye
* 00:14 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 00:08 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 00:01 wm-bot2: Upgraded and rebooted host cloudnet2006-dev.codfw.wmnet - cookbook ran by andrew@bullseye

=== 2023-01-05 ===
* 23:54 wm-bot2: Upgraded and rebooted host cloudnet2005-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 23:38 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 23:25 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 23:13 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 22:18 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 22:10 andrewbogott: upgrading codfw1dev openstack to version 'zed'

=== 2023-01-04 ===
* 21:18 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org - cookbook ran by andrew@bullseye
* 21:11 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org - cookbook ran by andrew@bullseye
* 20:12 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org - cookbook ran by andrew@bullseye
* 20:04 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org - cookbook ran by andrew@bullseye
* 14:45 wm-bot2: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] - cookbook ran by fran@wmf3169
* 14:44 wm-bot2: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:41 wm-bot2: Rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:41 wm-bot2: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:38 wm-bot2: Rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:37 wm-bot2: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev - cookbook ran by fran@wmf3169

=== 2023-01-03 ===
* 22:11 wm-bot2: Upgraded and rebooted host cloudservices2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Upgraded and rebooted host cloudservices2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 15:25 taavi: restart designate-sink everywhere to pick up wmf-sink changes

=== 2022-12-25 ===
* 14:21 taavi: register developer account 'instance-puppet-user-dev' to update the codfw1dev instance-puppet repo without access to the eqiad1 repo [[phab:T318504|T318504]]

=== 2022-12-22 ===
* 15:16 dcaro: added submit rights for JenkinsBot on all cloud/* gerrit repos

=== 2022-12-21 ===
* 04:59 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:59 wm-bot2: Finished rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:35 wm-bot2: Rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:35 wm-bot2: Finished rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:12 wm-bot2: Rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:12 wm-bot2: Finished rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:48 wm-bot2: Rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:48 wm-bot2: Finished rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:24 wm-bot2: Rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:24 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:00 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:00 wm-bot2: Finished rebooting node cloudcephosd1024.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:36 wm-bot2: Rebooting node cloudcephosd1024.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:36 wm-bot2: Finished rebooting node cloudcephosd1023.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:12 wm-bot2: Rebooting node cloudcephosd1023.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:12 wm-bot2: Finished rebooting node cloudcephosd1022.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:47 wm-bot2: Rebooting node cloudcephosd1022.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:47 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:22 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:22 wm-bot2: Finished rebooting node cloudcephosd1020.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:58 wm-bot2: Rebooting node cloudcephosd1020.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:58 wm-bot2: Finished rebooting node cloudcephosd1019.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:34 wm-bot2: Rebooting node cloudcephosd1019.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:34 wm-bot2: Finished rebooting node cloudcephosd1018.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:09 wm-bot2: Rebooting node cloudcephosd1018.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:09 wm-bot2: Finished rebooting node cloudcephosd1017.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye

=== 2022-12-20 ===
* 23:45 wm-bot2: Rebooting node cloudcephosd1017.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:45 wm-bot2: Finished rebooting node cloudcephosd1016.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:21 wm-bot2: Rebooting node cloudcephosd1016.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:21 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:56 wm-bot2: Rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:55 wm-bot2: Finished rebooting node cloudcephosd1014.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Rebooting node cloudcephosd1014.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Finished rebooting node cloudcephosd1013.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:08 andrewbogott: restarting openstack services with wmcs.openstack.restart_openstack due to miscellaneous trove failures
* 22:06 wm-bot2: Rebooting node cloudcephosd1013.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:06 wm-bot2: Finished rebooting node cloudcephosd1012.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Rebooting node cloudcephosd1012.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Finished rebooting node cloudcephosd1011.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Rebooting node cloudcephosd1011.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Finished rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Finished rebooting node cloudcephosd1009.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:26 wm-bot2: Rebooting node cloudcephosd1009.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:26 wm-bot2: Finished rebooting node cloudcephosd1008.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:01 wm-bot2: Rebooting node cloudcephosd1008.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:01 wm-bot2: Finished rebooting node cloudcephosd1007.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Rebooting node cloudcephosd1007.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Finished rebooting node cloudcephosd1006.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Rebooting node cloudcephosd1006.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Finished rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:45 wm-bot2: Rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:45 wm-bot2: Finished rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:20 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:20 wm-bot2: Finished rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:56 wm-bot2: Rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:56 wm-bot2: Finished rebooting node cloudcephosd1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Rebooting node cloudcephosd1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Finished rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:07 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:07 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 16:49 wm-bot2: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:48 wm-bot2: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:46 wm-bot2: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:46 wm-bot2: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 16:39 wm-bot2: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:39 wm-bot2: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:41 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:41 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 15:40 wm-bot2: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:40 wm-bot2: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:37 wm-bot2: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:37 wm-bot2: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:30 wm-bot2: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:30 wm-bot2: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:29 wm-bot2: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:29 wm-bot2: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:25 wm-bot2: Rebooting node cloudcephmon2006-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:25 wm-bot2: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Rebooting node cloudcephmon2005-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:19 wm-bot2: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:19 wm-bot2: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:18 wm-bot2: Finished rebooting the nodes ['cloudcephmon1001', 'cloudcephmon1002', 'cloudcephmon1003'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:18 wm-bot2: Finished rebooting node cloudcephmon1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:15 wm-bot2: Rebooting node cloudcephmon1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:15 wm-bot2: Finished rebooting node cloudcephmon1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Finished rebooting node cloudcephmon1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:08 wm-bot2: Rebooting node cloudcephmon1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:08 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye

=== 2022-12-17 ===
* 07:50 taavi: deleted project packagist-mirror per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge#packagist-mirror

=== 2022-12-16 ===
* 19:36 volans: restarted sshd twice on bastion-restricted-eqiad1-02 to debug SSH connections for [[phab:T319401|T319401]]
* 08:46 dcaro: restart designate-sink on both cloudservice hosts ([[phab:T322279|T322279]])
* 08:45 dcaro: restart designate-sink on both cloudservice hosts

=== 2022-12-07 ===
* 22:07 andrewbogott: systemctl restart libvirt-guests.service  on cloudvirt1019 to get ceph/rbd working on VMS on this hypervisor

=== 2022-12-03 ===
* 19:24 taavi: restart designate-sink on both cloudservices hosts

=== 2022-11-30 ===
* 20:03 andrewbogott: changing all rabbitmq queues to quorum queues.  Will be noisy!  [[phab:T318816|T318816]]
* 02:54 wm-bot2: Upgraded and rebooted host cloudbackup2002.codfw.wmnet - cookbook ran by andrew@bullseye

=== 2022-11-28 ===
* 13:00 wm-bot2: unset cloudvirt1043.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 10:28 wm-bot2: Drained cloudvirt1043.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:19 wm-bot2: Set cloudvirt cloudvirt1043.eqiad.wmnet maintenance (downtime id: bb94dd24-fef9-4c9c-8f79-{{Gerrit|b6e15023ce69}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:18 wm-bot2: Draining cloudvirt1043.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-25 ===
* 10:54 wm-bot2: deleted VM canary2001-dev-2 from cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 10:54 wm-bot2: created VM canary2001-dev-3 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 10:53 wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:5b2ca632-2ea0-4007-9b40-{{Gerrit|4f84f8e2428b}}) - cookbook ran by arturo@nostromo
* 10:46 wm-bot2: Created new flavor: g3.cores1.ram1.disk20.admin (id:fecbd56d-0969-45f3-80fd-{{Gerrit|2b463a5b6270}}) - cookbook ran by arturo@nostromo

=== 2022-11-24 ===
* 16:53 wm-bot2: deleted VM canary2001-dev-1 from cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:53 wm-bot2: created VM canary2001-dev-2 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2003-dev-1 in cloudvirt2003-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2002-dev-1 in cloudvirt2002-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2001-dev-1 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:36 wm-bot2: Created new flavor: cloudvirt-canary-ceph (id:0d06701a-2845-4298-b2b4-{{Gerrit|fabf8b1ddcbb}}) - cookbook ran by arturo@nostromo
* 13:03 wm-bot2: unset cloudvirt1044.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 11:51 wm-bot2: Drained cloudvirt1044.eqiad.wmnet - cookbook ran by arturo@nostromo
* 11:37 wm-bot2: Set cloudvirt cloudvirt1044.eqiad.wmnet maintenance (downtime id: 10076ecc-0f94-4d56-9bbd-{{Gerrit|bebb48bdc126}}, use this to unset) - cookbook ran by arturo@nostromo
* 11:35 wm-bot2: Draining cloudvirt1044.eqiad.wmnet - cookbook ran by arturo@nostromo
* 10:16 dcaro: removed ip6 dns name entry from nb for coluddb* ([[phab:T323550|T323550]])
* 09:53 dcaro: removed ip6 dns entry from nb for coluddb1013 ([[phab:T323550|T323550]])

=== 2022-11-23 ===
* 15:00 wm-bot2: unset cloudvirt1045.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Drained cloudvirt1045.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:35 wm-bot2: Set cloudvirt cloudvirt1045.eqiad.wmnet maintenance (downtime id: 2386b468-0f21-4ecb-91e2-{{Gerrit|e19ace66881d}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:34 wm-bot2: Draining cloudvirt1045.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:20 wm-bot2: unset cloudvirt1046.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 12:17 wm-bot2: Drained cloudvirt1046.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:04 wm-bot2: Set cloudvirt cloudvirt1046.eqiad.wmnet maintenance (downtime id: 6291d38e-c04c-4aa0-88da-{{Gerrit|2b329874a9b9}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:03 wm-bot2: Draining cloudvirt1046.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:02 wm-bot2: unset cloudvirt1047.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 11:32 arturo: [codfw1dev] created project cloudvirt-canary
* 10:13 wm-bot2: Drained cloudvirt1047.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:01 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 2c7ccc17-2be3-427d-aaec-{{Gerrit|57fadca0de5b}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:00 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-22 ===
* 13:26 wm-bot2: unset cloudvirt1048.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 12:15 wm-bot2: unset cloudvirt1049.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 10:58 wm-bot2: Drained cloudvirt1049.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:37 wm-bot2: Set cloudvirt cloudvirt1049.eqiad.wmnet maintenance (downtime id: 3ae0a20d-3cf0-4eba-bea6-{{Gerrit|45aa61d8ad00}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:36 wm-bot2: Draining cloudvirt1049.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:21 wm-bot2: Unset cloudvirt cloudvirt1050.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo

=== 2022-11-21 ===
* 16:19 wm-bot2: Drained cloudvirt1050.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:11 wm-bot2: Set cloudvirt cloudvirt1050.eqiad.wmnet maintenance (downtime id: 0b154a93-a9d3-4ac7-bcfc-{{Gerrit|b67c49abe97b}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:09 wm-bot2: Draining cloudvirt1050.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:07 wm-bot2: Unset cloudvirt cloudvirt1051.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 15:18 wm-bot2: Drained cloudvirt1051.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 15:02 wm-bot2: Set cloudvirt cloudvirt1051.eqiad.wmnet maintenance (downtime id: 1de1174b-ec46-47a3-911c-{{Gerrit|b5808ce37028}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 15:00 wm-bot2: Draining cloudvirt1051.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 14:57 wm-bot2: Unset cloudvirt cloudvirt1052.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 12:53 wm-bot2: Drained cloudvirt1052.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:30 wm-bot2: Set cloudvirt cloudvirt1052.eqiad.wmnet maintenance (downtime id: 30db8a9b-08db-456f-8106-{{Gerrit|53188ff5f989}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:29 wm-bot2: Draining cloudvirt1052.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:15 wm-bot2: Unset cloudvirt cloudvirt1053.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 10:13 arturo: drained cloudvirt1053 in preparation for reimage (was spare anyway)

=== 2022-11-18 ===
* 13:37 arturo: [codfw1dev] reimaged cloudvirt2001-dev and cloudvirt2002-dev
* 11:36 wm-bot2: Set cloudvirt cloudvirt2001-dev.codfw.wmnet maintenance (downtime id: 2fb9df34-fc2d-45b9-b21f-{{Gerrit|c0ec09008b92}}, use this to unset) - cookbook ran by arturo@nostromo
* 11:36 wm-bot2: Draining cloudvirt2001-dev.codfw.wmnet - cookbook ran by arturo@nostromo

=== 2022-11-16 ===
* 20:07 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:56 wm-bot2: Upgraded and rebooted host cloudservices2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:16 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 18:59 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 12:52 arturo: failovered cloudgw1002 into cloudgw1001 for reimage, IRC bots were briefly disconnected

=== 2022-11-14 ===
* 20:22 wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye
* 20:07 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 19:42 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye
* 19:28 andrewbogott: beginning OpenStack upgrade in eqiad1 -- [[phab:T305828|T305828]]
* 19:22 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 19:14 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 19:08 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 12:52 arturo: cleanup old network vlan interface names from /etc/network/interfaces in cloudnet1005/1006

=== 2022-11-11 ===
* 13:24 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: edad3915-b7c6-4b23-bb9c-{{Gerrit|ab13b04a41c5}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:23 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:17 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: a09a9868-8aae-4dc0-8510-{{Gerrit|a3923b703060}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:16 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-10 ===
* 16:19 wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: 346013ec-ce4e-497e-ad65-{{Gerrit|2d215b14998c}}, use this to unset). ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:18 wm-bot2: Draining 'cloudvirt2002-dev.codfw.wmnet'. ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:13 wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: a40a8487-22ee-4bc6-bbe4-{{Gerrit|694a615e3bf5}}, use this to unset). ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-08 ===
* 11:17 taavi: backfilling security groups for metricsinfra access on all projects [[phab:T288108|T288108]]

=== 2022-11-07 ===
* 21:01 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 20:40 andrewbogott: upgrading eqiad1 designate to version 'yoga'

=== 2022-11-04 ===
* 17:57 andrewbogott: removing cinderv2 API endpoints from keystone catalog; this is deprecated and removed in Yoga.  prep for [[phab:T305828|T305828]]

=== 2022-11-03 ===
* 19:24 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 00:08 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye

=== 2022-11-02 ===
* 23:39 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:38 wm-bot2: Upgraded and rebooted host cloudnet2006-dev.codfw.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Upgraded and rebooted host cloudnet2005-dev.codfw.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:13 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:00 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 22:54 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 20:29 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye

=== 2022-10-31 ===
* 13:09 arturo: restart keepalived on all 4 cloudgw servers to run them with `-D` in /etc/default/keepalived to further debug [[phab:T320975|T320975]]

=== 2022-10-26 ===
* 16:18 wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:bf48880d-0c1b-4c2a-8e8b-{{Gerrit|778d28b16561}}) ([[phab:T319446|T319446]]) - cookbook ran by dcaro@vulcanus
* 09:34 taavi: running wmcs-puppetcertleaks in delete mode
* 09:09 taavi: running wmcs-novastats-dnsleaks in delete mode

=== 2022-10-25 ===
* 16:03 arturo: [codfw1dev] [[phab:T321220|T321220]] root@cloudcontrol2001-dev:~# openstack subnet create magnum --no-dhcp --network 57017d7c-3817-429a-8aa3-{{Gerrit|b028de82cdcc}} --ip-version 4 --gateway auto --subnet-range 192.168.0.0/24
* 14:38 arturo: [codfw1dev] restart neutron-l3-agent in cloudnet2005-dev, it was dead after rabbit connectivity problems

=== 2022-10-24 ===
* 18:50 wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Finished rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:47 wm-bot2: Rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:47 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by andrew@bullseye
* 18:38 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by andrew@bullseye
* 18:21 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:21 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl

=== 2022-10-20 ===
* 23:23 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:23 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:20 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:20 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:19 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:16 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:07 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:07 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:06 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:06 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 101c41d1-d65d-4088-b6d2-{{Gerrit|eac859e45ef8}}, use this to unset). - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:01 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:57 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 7197ce34-cc57-4677-a1a9-{{Gerrit|05e20cd0dd80}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:38 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 68e4cc1d-9def-444e-84a7-{{Gerrit|21b0b5adfa72}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 73388c1e-369b-40af-a2c1-{{Gerrit|96936128c324}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 12feeca2-ea59-48e9-9235-{{Gerrit|1cecf5b384cd}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 1b34587f-2770-4d2e-bb31-{{Gerrit|c8ad14632d39}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:24 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:13 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: e75b00eb-7f58-4821-8139-{{Gerrit|3dfc6e97a92a}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 9151511e-bcc0-4367-a976-{{Gerrit|7cff060308aa}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 241c0bd9-c3cf-40e4-95dc-{{Gerrit|9b51d9823fe9}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 7a53c274-1d5f-4c94-9a0d-{{Gerrit|721c3e8f7239}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:10 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:10 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:06 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:54 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:54 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:51 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 0b75b44c-1efe-4edf-8f5e-{{Gerrit|42a67e8d3b13}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: aeb9c3a6-961a-4873-85d4-{{Gerrit|929248aebb8b}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: eb14cc04-3293-4cf8-a46f-{{Gerrit|1f87d8b0bcc4}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 84a3cd99-2643-495a-86b6-{{Gerrit|7ae80b11a30d}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:32 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:32 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:28 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:28 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 01a0b69c-2e27-4331-9635-{{Gerrit|403a668dac29}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:27 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:27 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:23 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: ea29f65f-6d86-4568-8db9-{{Gerrit|a85faa827447}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:22 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:22 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:18 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:18 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:16 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:14 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: d6b6bb4d-c7c0-4bd2-9a02-{{Gerrit|dee9a5ec6a3e}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 4e109149-7d67-403f-8b21-{{Gerrit|f829235ea491}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: 341f4b8c-6d6e-4190-9f2d-{{Gerrit|a1a1483abadd}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: f76e8f14-920e-4d51-a44e-{{Gerrit|321a55dcb0c5}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:56 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:56 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:54 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:35 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:34 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 0b0d8090-4d31-4d43-8545-{{Gerrit|c25209e2ef58}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:31 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 3d928554-a79c-419a-bcc4-{{Gerrit|c8d63791d8e7}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 3d3c8aa5-45f7-429e-a9d7-{{Gerrit|b5181b29dc13}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: b9027020-8a90-4c40-b0e6-{{Gerrit|6c8767f87917}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: f0d60fab-64e9-4da2-826c-{{Gerrit|229d31d0fbc2}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:16 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:14 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:11 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: b65da7e2-9fd2-4a1e-8dd4-{{Gerrit|88ca65936ae4}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: cbbf114c-9a4c-4dd2-9b58-{{Gerrit|50da8bd896ca}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 5969beaf-72bf-4af9-ae4d-{{Gerrit|8e5c3331e78e}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: b8f7389d-79b4-411d-b3d5-{{Gerrit|ec8f92eba101}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:53 wm-bot2: Safe reboot of 'cloudvirt1051.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:53 wm-bot2: Unset cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:51 wm-bot2: Safe reboot of 'cloudvirt1049.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:51 wm-bot2: Unset cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:50 wm-bot2: Safe reboot of 'cloudvirt1048.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:50 wm-bot2: Unset cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Drained 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: c5dbfa7b-72fc-4156-8257-{{Gerrit|af224a725b78}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: c1a3e92c-cb04-46e4-ac5d-{{Gerrit|784c79601b05}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance (downtime id: 62fdc4ee-c8d5-4892-aeb9-{{Gerrit|a681cdcbc84b}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 9ec407cd-5c00-407a-a57d-{{Gerrit|794f6c68f947}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Set cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance (downtime id: 712683ea-d58f-484b-8efb-{{Gerrit|89d1c21aa0e3}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Safe rebooting 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Draining 'cloudvirt1049.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Safe rebooting 'cloudvirt1049.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Draining 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Safe rebooting 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:25 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:25 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:21 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:04 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 18fae8d8-7353-4f67-90d7-{{Gerrit|8df9b3fb1ccb}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:03 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:03 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:01 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 3d3cffa3-abc6-4901-83d9-{{Gerrit|3bcc4b02fd3c}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:00 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:00 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:54 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:54 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:46 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:46 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye

=== 2022-10-15 ===
* 17:38 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki Slevinski # [[phab:T320867|T320867]]

=== 2022-10-13 ===
* 12:19 wm-bot2: OSDs (['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034'] ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 11:30 wm-bot2: OSDs (['cloudcephosd1025', 'cloudcephosd1026']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus

=== 2022-10-10 ===
* 14:01 dcaro: test2
* 01:22 andrewbogott: restarting designate-sink on cloudservices100[45], possible example of [[phab:T316614|T316614]]

=== 2022-10-09 ===
* 12:04 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki DatGuy # [[phab:T320301|T320301]]

=== 2022-10-07 ===
* 13:40 andrewbogott: dhinus is resetting rabbitmq cluster in an attempt to resolve a suspected (by Andrew) split-brain
* 11:33 arturo: rabbitmq-server.service @ cloudrabbit1002 is again up and running ([[phab:T320232|T320232]])
* 10:24 arturo: stopping rabbitmq-server.service @ cloudrabbit1002 ([[phab:T320232|T320232]])
* 10:19 arturo: restarting nova-conductor in all 3 cloudcontrols ([[phab:T320232|T320232]])
* 09:45 arturo: restarting rabbitmq-server.service @ cloudrabbit1002 ([[phab:T320232|T320232]])

=== 2022-10-06 ===
* 15:55 arturo: cloudnet1005 &amp; cloudnet1006 now in service. Secom cloudnet1003 &amp; cloudnet1004. Drop neutron agents, etc. ([[phab:T316284|T316284]])
* 11:54 arturo: rebooting cloudnet1005/1006 to see if they have the right network config ([[phab:T316284|T316284]])
* 11:50 arturo: set neutron l3 agents on cloudnet1005/1006 as down `root@cloudcontrol1005:~# neutron agent-update --admin-state-down &lt;uuid&gt;` ([[phab:T316284|T316284]])
* 11:40 arturo: [codfw1dev] rebooting both network nodes to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/839492
* 10:14 arturo: [codfw1dev] restart neutron-l3-agent on cloudnet2006-dev, it was dead

=== 2022-10-05 ===
* 14:40 wm-bot2: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T319418|T319418]]) - cookbook ran by fran@wmf3169
* 14:40 wm-bot2: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T319418|T319418]]) - cookbook ran by fran@wmf3169
* 14:28 arturo: adding cloudinstances2b-gw router to l3 agents on cloudnet1005/1006 ([[phab:T316284|T316284]])
* 13:11 wm-bot2: Added 1 new OSDs ['cloudcephosd1034.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 13:11 wm-bot2: Added OSD cloudcephosd1034.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 13:02 wm-bot2: Finished rebooting node cloudcephosd1034.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Rebooting node cloudcephosd1034.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Adding OSD cloudcephosd1034.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Adding new OSDs ['cloudcephosd1034.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169

=== 2022-10-04 ===
* 16:40 wm-bot2: Added 1 new OSDs ['cloudcephosd1033.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 16:40 wm-bot2: Added OSD cloudcephosd1033.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:20 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 14:17 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:59 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:56 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:55 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:55 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169

=== 2022-09-30 ===
* 14:52 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169
* 14:52 wm-bot2: Added OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:48 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:48 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 64eac5c6-4b1d-4269-98fd-{{Gerrit|8e5bed42ce40}}, use this to unset). ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:09 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:05 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: fb99c967-b974-4314-a2fa-{{Gerrit|31ed0e883dd3}}, use this to unset). - cookbook ran by andrew@buster
* 14:04 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 13:30 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169
* 13:30 wm-bot2: Added OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:26 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:26 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 13:16 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:16 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 13:08 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:08 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 12:45 wm-bot2: Finished rebooting node cloudcephosd1031.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:42 wm-bot2: Rebooting node cloudcephosd1031.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:41 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:41 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:26 arturo: [codfw1dev] cloudnet2005/2006-dev are now on a single NIC setup ([[phab:T318824|T318824]])
* 11:44 arturo: sysctl change (cleanup) on cloudnet1003/1004

=== 2022-09-27 ===
* 10:48 wm-bot2: Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:35 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:26 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:23 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:22 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:22 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:17 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:09 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169

=== 2022-09-26 ===
* 18:37 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 18:37 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:33 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:31 andrewbogott: rebooting cloudvirt1028 for [[phab:T317391|T317391]]
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance (downtime id: 6d1c4c53-76b9-493f-8c44-{{Gerrit|eb0413dfb9d0}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance (downtime id: 6bb80b65-616c-4e45-be4f-{{Gerrit|d2cdd8abb2bf}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance (downtime id: a3bba7e7-bcd3-482d-a486-{{Gerrit|06b6f53fd899}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:18 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8fb59651-fd42-4aee-a978-{{Gerrit|36bc7f79b4d5}}, use this to unset). - cookbook ran by andrew@buster
* 18:18 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:17 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:01 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:01 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:58 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:51 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:51 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:47 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:47 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:47 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:45 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:38 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: abfc35bb-331e-4d7e-bb92-{{Gerrit|35e675abce32}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:34 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:33 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: d4fe6ea8-76eb-45f7-b6cf-{{Gerrit|66f54ba9801c}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:33 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: c0671bda-9aae-4960-85be-{{Gerrit|58aaa9c8e021}}, use this to unset). - cookbook ran by andrew@buster
* 17:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:32 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:32 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:28 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:28 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:24 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: be6559f9-4d72-4492-b9b9-{{Gerrit|23c636d6554e}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:23 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:23 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:21 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 237dfb1b-5382-408d-aa35-{{Gerrit|1aa1cfe4c5af}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 088b97cd-0cf2-4c27-a67f-{{Gerrit|077ffa1c1c5e}}, use this to unset). - cookbook ran by andrew@buster
* 17:16 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:16 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 91625000-aa6d-4887-bf34-{{Gerrit|ef7785a4af4a}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:16 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:15 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:15 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:15 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:15 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 7364c4fe-9f6b-4539-b6fa-{{Gerrit|1e768b602a1b}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:14 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:14 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:12 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:12 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: cb8f8a21-73c0-4f54-9412-{{Gerrit|074d82582cb4}}, use this to unset). - cookbook ran by andrew@buster
* 17:11 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:11 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:09 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: d23dabd2-3bfc-4ce0-9533-{{Gerrit|cd1cf910f8e1}}, use this to unset). - cookbook ran by andrew@buster
* 17:08 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:08 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:05 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 4d7ddae1-f621-4dd5-8616-{{Gerrit|29b7699b692f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 4628dd6f-5b49-417f-b6ab-{{Gerrit|5c41992192b4}}, use this to unset). - cookbook ran by andrew@buster
* 17:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: c95529cf-b2a5-4553-bd6e-{{Gerrit|6ad4dcb2a105}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:01 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 36c68112-a964-4a7c-a093-{{Gerrit|8a6d481dea7d}}, use this to unset). - cookbook ran by andrew@buster
* 17:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:00 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:00 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:49 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:49 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:45 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:31 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 1244c159-65bb-476e-a702-{{Gerrit|8f43c253e499}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:30 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:30 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:22 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 13:19 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 13:18 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:18 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 12:32 dcaro: Changed the collation of labsdbaccount db to utf8mb4_bin ([[phab:T318047|T318047]])
* 10:14 arturo: deployed new version of maintain-dbusers ([[phab:T318047|T318047]])

=== 2022-09-25 ===
* 15:06 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:06 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: a050e47f-3a63-41ff-accb-{{Gerrit|3993c1e8b593}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:05 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:05 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:58 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:58 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 1a49d773-4dc9-4a6f-bd49-{{Gerrit|8663db77ce66}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:57 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:57 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:54 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: ab52dd42-68a6-4159-b345-{{Gerrit|e5625d209b57}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:53 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:53 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 10337415-3095-4834-8538-{{Gerrit|b3a64bd6b18d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:53 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:53 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 4cdbed3a-3775-4913-8c3b-{{Gerrit|afd57ae1ca55}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-24 ===
* 17:37 andrewbogott: restarting neutron api on cloudcontrol1006; cause of outage unknown
* 17:35 andrewbogott: restarting neutron-linuxbridge-agent on cloudvirt1022

=== 2022-09-22 ===
* 15:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:10 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04849437-a304-45a1-a037-{{Gerrit|538741a2f801}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:09 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:09 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:07 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:07 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:04 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 8810a919-61e7-4ba5-af7f-{{Gerrit|c41d1e1c8c8b}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:59 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:59 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:56 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: b6f8e0b6-f35c-41fd-8035-{{Gerrit|6efde61db3a3}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 784bff2c-8160-44dc-9bcf-{{Gerrit|ff23055a0527}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:40 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:40 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:39 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:38 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:38 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 21152e9e-bc67-4024-b68e-{{Gerrit|fd671c398642}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: f07ddb1d-e8da-43f3-8140-{{Gerrit|bc66789d37c8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:36 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:36 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:33 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:33 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:30 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 4e4b03fc-e3f1-440e-9097-{{Gerrit|cce5edaa1f08}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: d4bb2160-f492-4ddc-a5b7-{{Gerrit|eeee37007d14}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9bf00a6a-3697-4201-9a6e-{{Gerrit|76d499eccfa1}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:13 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:13 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:47 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f2888490-1804-4b23-b7b0-{{Gerrit|677853fb9ef7}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8a3e8b0a-ced3-4667-a121-{{Gerrit|66df673c7ed8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:43 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 35c761b3-f0de-4b23-9b11-{{Gerrit|2ed2e474c89f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 6121a0e1-754a-47fa-b51b-{{Gerrit|265f6386f396}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:40 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:40 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:39 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 189ae1bb-c69d-4eb0-aee9-{{Gerrit|90ab1f492aa8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:38 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:38 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:36 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: acc53776-68bf-47a4-bb82-{{Gerrit|c571b3df2945}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:31 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 07770ae7-3e7e-4615-8174-{{Gerrit|513767f95b1d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:30 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:30 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 7ae20a7d-8bff-4bc2-91c5-{{Gerrit|b0005c8164d1}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: b596f855-8019-49d2-9657-{{Gerrit|0d513ad7acb5}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:25 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 9172c785-d2d6-46e5-bed3-{{Gerrit|2f49d1033f78}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:23 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:23 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:19 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 85b4e0e2-e635-485f-9ce5-{{Gerrit|341981b3887f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 9494c283-02c8-42da-98c3-{{Gerrit|139b4d119ef8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: d181f301-249f-4b5a-bd85-{{Gerrit|47d87262d0ed}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-20 ===
* 21:02 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 21:02 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:58 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:58 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 1c4246a4-9cee-4423-8cd1-{{Gerrit|4f52f61d503f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:53 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:50 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:50 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:49 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 82665ecc-4431-48fe-b255-{{Gerrit|7e9d518be217}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:48 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:48 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:47 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: cd284562-e3b0-4504-9977-{{Gerrit|2b18032bbf3c}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:45 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 8fb2a646-16a5-4183-94a1-{{Gerrit|ee6bbbf029fa}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:44 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:44 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: d561fe45-1582-4cd8-bbc9-{{Gerrit|a356c39f3330}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 351c365c-0228-4907-a279-{{Gerrit|01795b1e62ce}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 3beec1dd-0132-4f5c-adce-{{Gerrit|5a7dc56060b6}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:30 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:29 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:28 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:28 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:26 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:24 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:20 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:16 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 353a8527-ad5e-4898-937c-{{Gerrit|303d16801e28}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: fc235146-f070-4723-9503-{{Gerrit|e20cbb877755}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: cdf712ac-c083-4a63-af83-{{Gerrit|514ca5cdc76d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:05 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:01 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:01 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 8f2d2d32-e99e-4e2c-9472-{{Gerrit|ca33b577d01f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 66ee7d87-cee1-4c8f-b5c3-{{Gerrit|5f58a6cea336}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:59 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:59 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:58 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:58 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:58 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 69035b38-9910-46f1-9582-{{Gerrit|3105c1ba4d16}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:57 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:57 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:52 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 5a94c9d1-c956-4b03-a92b-{{Gerrit|b112810906a4}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:51 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:51 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:50 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 216a9120-4b9a-4be3-99db-{{Gerrit|3fd5fdc25146}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:49 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:49 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 46fb1964-3cf4-47fd-ad79-{{Gerrit|475f33c8ed38}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: b04a4f8c-db82-4258-8608-{{Gerrit|2a78ec9193ed}}, use this to unset). - cookbook ran by andrew@buster
* 19:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 7ae7c258-ecb8-47d1-a991-{{Gerrit|3fa617e305bf}}, use this to unset). - cookbook ran by andrew@buster
* 19:45 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 377fb0e5-132e-4e7a-9079-{{Gerrit|37937c3d42bf}}, use this to unset). - cookbook ran by andrew@buster
* 19:44 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 6c4222ae-796e-4890-8820-{{Gerrit|9e7ce5438f2a}}, use this to unset). - cookbook ran by andrew@buster
* 19:43 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:43 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:38 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:34 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:34 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:31 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: eff98e2e-ae99-41d0-a68e-{{Gerrit|671d941fcf21}}, use this to unset). - cookbook ran by andrew@buster
* 19:30 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5310f1eb-0425-4f59-8fb5-{{Gerrit|1d9c6f6d2f23}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:29 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:25 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: d8ab4682-026a-4ddc-bfc5-{{Gerrit|48166bd9789a}}, use this to unset). - cookbook ran by andrew@buster
* 19:24 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:23 wm-bot2: Safe reboot of 'cloudvirt1048.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:23 wm-bot2: Unset cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: eb2f9d94-8ea5-48d5-a336-{{Gerrit|97dd407b1ce3}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Safe rebooting 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:18 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:14 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 305fe2d7-913e-4638-95c2-{{Gerrit|4e810dc6b2a3}}, use this to unset). - cookbook ran by andrew@buster
* 19:14 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5bb3a66b-9450-4230-8d09-{{Gerrit|d78473d72ba2}}, use this to unset). - cookbook ran by andrew@buster
* 19:13 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:13 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:07 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: b1e39b49-3a88-4e42-913c-{{Gerrit|9892eafad447}}, use this to unset). - cookbook ran by andrew@buster
* 19:06 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:05 andrewbogott: putting cloudvirt1049-1052 into 'ceph' pool, taking out of 'spare' pool. cloudvirt1053 will remain our only spare.
* 19:02 wm-bot2: Safe reboot of 'cloudvirt1049.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:02 wm-bot2: Unset cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:00 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:00 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:59 wm-bot2: Safe reboot of 'cloudvirt1051.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:59 wm-bot2: Unset cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Drained 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Set cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance (downtime id: 8f24ea75-e107-4a13-8bd6-{{Gerrit|950b941b8e03}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Draining 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Safe rebooting 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:57 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:57 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:56 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:56 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 5c3e8fa4-cd1e-43f4-a423-{{Gerrit|ba7fe2459d4e}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Drained 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Set cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance (downtime id: e3fe22d6-932c-4502-8838-{{Gerrit|82cdc4c8f1c6}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:54 wm-bot2: Draining 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:54 wm-bot2: Safe rebooting 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:53 wm-bot2: Safe reboot of 'cloudvirt1053.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:53 wm-bot2: Unset cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: fd824b92-b6ff-4631-a2d4-{{Gerrit|debb25d48223}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:51 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:49 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 52d5cd5b-0f56-453a-85f8-{{Gerrit|0b5e656ae7f9}}, use this to unset). - cookbook ran by andrew@buster
* 18:49 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:49 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: b92f66a3-a487-4515-b68b-{{Gerrit|1c257a57b893}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:48 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-19 ===
* 20:07 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 4bfca6cd-dfbc-4a79-9203-{{Gerrit|b432bfeee5c2}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:41 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:21 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 70479325-609b-4349-9094-{{Gerrit|739eb9b3bb30}}, use this to unset). - cookbook ran by andrew@buster
* 19:21 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-09-14 ===
* 16:57 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
* 11:01 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:58 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:57 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:57 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:09 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:09 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:06 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:06 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 09:46 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-09-13 ===
* 12:16 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:16 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:11 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:11 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:36 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-09-10 ===
* 15:37 andrewbogott: restarting nova-conductor service (and possibly others, in response to lots of unanswered rabbitmq messages)

=== 2022-09-08 ===
* 19:12 andrewbogott: restarting nginx on proxy-03.project-proxy.eqiad1.wikimedia.cloud

=== 2022-09-07 ===
* 10:18 wm-bot2: Added OSD cloudcephosd1032.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:14 wm-bot2: Finished rebooting node cloudcephosd1032.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Rebooting node cloudcephosd1032.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Adding OSD cloudcephosd1032.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Adding new OSDs ['cloudcephosd1032.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:41 dhinus: Temporarily removing cloudcephosd1030 from Ceph cluster (https://phabricator.wikimedia.org/T314870)

=== 2022-08-30 ===
* 14:59 andrewbogott: manually marking most eqiad1 cloud* servers down in icinga for [[phab:T296561|T296561]]
* 10:43 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:39 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:38 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:38 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:39 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:14 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:11 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-25 ===
* 15:14 wm-bot2: Added 1 new OSDs ['cloudcephosd1029.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 15:14 wm-bot2: Added OSD cloudcephosd1029.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 15:02 wm-bot2: Finished rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:59 wm-bot2: Rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:58 wm-bot2: Adding OSD cloudcephosd1029.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:58 wm-bot2: Adding new OSDs ['cloudcephosd1029.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-24 ===
* 22:07 andrewbogott: replaced cloudservices1003 with cloudservices1005 [[phab:T304888|T304888]]
* 10:45 wm-bot2: Added 1 new OSDs ['cloudcephosd1028.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:45 wm-bot2: Added OSD cloudcephosd1028.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Finished rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:34 wm-bot2: Rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:33 wm-bot2: Adding OSD cloudcephosd1028.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:33 wm-bot2: Adding new OSDs ['cloudcephosd1028.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-23 ===
* 13:46 wm-bot2: Added 1 new OSDs ['cloudcephosd1027.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:46 wm-bot2: Added OSD cloudcephosd1027.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:27 wm-bot2: Finished rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:24 wm-bot2: Rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:22 wm-bot2: Adding OSD cloudcephosd1027.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:22 wm-bot2: Adding new OSDs ['cloudcephosd1027.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-21 ===
* 21:12 andrewbogott: restarted neutron-dhcp-agent on cloudnet1003. it was claiming to be unable to contact Rabbit but seems happy after a restart

=== 2022-08-20 ===
* 07:39 dcaro_away: cloudvirt1023 is back up, VMs are starting to recover ([[phab:T315718|T315718]])
* 07:23 dcaro_away: cloudvirt1023 seems to have gotten some hardware issue from racadm lclog view &quot;System CPU Resetting.&quot;, rebooting and doing memory checks ([[phab:T315718|T315718]])

=== 2022-08-19 ===
* 17:06 taavi: [codfw1dev] restart mariadb on clouddb2002-dev to pick up certificate config changes [[phab:T310795|T310795]]

=== 2022-08-18 ===
* 13:25 wm-bot2: Added 1 new OSDs ['cloudcephosd1026.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:25 wm-bot2: Added OSD cloudcephosd1026.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:15 wm-bot2: Finished rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:12 wm-bot2: Rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:10 wm-bot2: Adding OSD cloudcephosd1026.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:10 wm-bot2: Adding new OSDs ['cloudcephosd1026.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 07:29 dcaro: Starting up all the osd daemons on cloudcephosd1025 ([[phab:T314870|T314870]])

=== 2022-08-17 ===
* 10:50 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:49 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:50 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:49 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:49 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:16 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:16 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:16 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-16 ===
* 22:39 andrewbogott: replacing the now-rebuilt cloudvirt1025 in 'ceph' aggregate and removing it from the 'maintenance' aggregate
* 17:41 andrewbogott: removing cloudvirt1025 from the 'ceph' aggregate and adding it to the 'maintenance' aggregate
* 17:40 andrewbogott: reimaging cloudvirt1025 after I accidentally deleted the hw raid
* 17:38 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1003@rbd#RBD --newhost cloudcontrol1005@rbd#RBD
* 17:37 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1004@rbd#RBD --newhost cloudcontrol1006@rbd#RBD
* 16:26 wm-bot2: Ceph cluster at eqiad1 set out of maintenance. - cookbook ran by dcaro@vulcanus
* 15:43 wm-bot2: Restarting the osd daemons from nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,c
* 15:42 wm-bot2: Finished restarting all the OSD daemons from the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
* 15:38 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:02 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 12:59 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus

=== 2022-08-14 ===
* 18:36 taavi: deleted the http keystone endpoints from the keystone service catalog

=== 2022-08-11 ===
* 13:57 andrewbogott: decommissioning cloudcontrol1003 + cloudcontrl1004. I backed up $home in case anyone needs their files.
* 08:42 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:42 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:42 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-10 ===
* 13:10 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-04 ===
* 17:16 taavi: deleted all scheduler_fanout_ rabbit queues in an attempt to fix scheduling
* 16:32 taavi: restart neutron-l3-agent to pick up rabbit config changes
* 15:12 andrewbogott: stopping rabbitmq on cloudcontrol1xxx
* 09:57 taavi: stop wikitech_run_jobs.timer on labweb1001/1002, hosts pending decom

=== 2022-08-03 ===
* 20:55 andrewbogott: root@tools-checker-04:~# systemctl restart uwsgi-toolschecker_cron.service
* 20:41 andrewbogott: restarting  neutron-l3-agent.service on cloudnet1003 and 1004. The agent was routing properly but had lost touch with rabbitmq

=== 2022-08-02 ===
* 14:07 andrewbogott: shutting down codfw1dev ceph cluster according to https://docs.mirantis.com/mcp/q4-18/mcp-operations-guide/scheduled-maintenance-power-outage/power-off-ceph-cluster.html
* 13:54 andrewbogott: shutting down basically all of codfw1dev to support pdu maintenance -- all the ceph OSDs will lose power so best to have everything stopped.

=== 2022-07-27 ===
* 19:32 andrewbogott: switching the openstack.eqiad1.wikimedia.cloud endpoint from cloudcontrol1004 to 1006, https://gerrit.wikimedia.org/r/c/operations/dns/+/817878/2/templates/wikimediacloud.org#54
* 16:33 andrewbogott: here is a test message in the admin channel

=== 2022-07-25 ===
* 13:43 andrewbogott: pooling cloudweb100[34] and depooling labweb100[12] for testing in prep for decomming labweb100[12]

=== 2022-07-22 ===
* 16:41 taavi: depool cloudweb1003/1004 since horizon seems to be having issues
* 16:22 taavi: pooling cloudweb1003/1004 now that grant issues are sorted

=== 2022-07-21 ===
* 18:26 andrewbogott: depooling cloudweb1003 and 1004 for wikitech, horizon, striker -- pending db grant changes
* 18:06 andrewbogott: pooling cloudweb1003 and 1004 for wikitech, horizon, striker

=== 2022-07-20 ===
* 18:02 dcaro: things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 ([[phab:T313400|T313400]])
* 17:45 bd808: `sudo service striker restart` on labweb1002
* 17:43 bd808: `sudo service striker restart` on labweb1001
* 17:10 dcaro: things seem stable, trying to bring up a fourth rabbit node, cloudcontrol1006 ([[phab:T313400|T313400]])
* 16:26 dcaro: things seem stable, trying to bring up a third, cloudcontrol1005 ([[phab:T313400|T313400]])
* 15:51 dcaro: things seem stable now with one rabbit node, trying to bring up a second ([[phab:T313400|T313400]])
* 14:16 dcaro: stopping rabbin on cloudcontrol1004, leaving only 1003 alive ([[phab:T313400|T313400]])
* 13:17 dcaro: restarting the whole rabbit cluster ([[phab:T313400|T313400]])

=== 2022-07-19 ===
* 16:30 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:30 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 16:26 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:18 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:18 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 16:14 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:01 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:01 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:57 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:57 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 3da2d4f6-5c5b-4a21-9a0b-{{Gerrit|2b010960ed6a}}, use this to unset). - cookbook ran by andrew@buster
* 15:56 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:49 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 8a10505a-107d-4e78-ba84-{{Gerrit|363a7dea8d69}}, use this to unset). - cookbook ran by andrew@buster
* 15:47 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 752aa58b-44d4-4340-b05d-{{Gerrit|911b14f2314d}}, use this to unset). - cookbook ran by andrew@buster
* 15:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: f89f0851-ce3f-4a92-899e-{{Gerrit|0d4638acc9c3}}, use this to unset). - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:45 andrewbogott: adding new hosts to the 'ceph' aggregate: cloudvirt1046, 1047, 1048
* 15:44 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:44 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:42 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:42 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:40 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:38 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:37 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:37 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:33 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:32 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 32d9fdb7-6c42-4cf4-9950-{{Gerrit|6e2442255161}}, use this to unset). - cookbook ran by andrew@buster
* 15:31 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:31 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:16 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 56ce1388-792c-442c-bb89-{{Gerrit|e4c3869b0acf}}, use this to unset). - cookbook ran by andrew@buster
* 15:15 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:15 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:14 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 3f84bf2e-24de-4ebe-8117-{{Gerrit|0f4a5de29d09}}, use this to unset). - cookbook ran by andrew@buster
* 15:14 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:14 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:13 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:13 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:12 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 5418e923-997c-49ae-b937-{{Gerrit|35fc0c3038eb}}, use this to unset). - cookbook ran by andrew@buster
* 15:12 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: c3dfdc2b-0725-4d01-87f2-{{Gerrit|440a5ac4694c}}, use this to unset). - cookbook ran by andrew@buster
* 15:11 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:09 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:04 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:04 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:00 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:54 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:54 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:50 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:46 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 49dbf7de-c58a-4b68-bc7b-{{Gerrit|f001a047f4c7}}, use this to unset). - cookbook ran by andrew@buster
* 14:46 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:46 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:44 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: a4e95168-39b5-452d-8fce-{{Gerrit|7875ae44a62b}}, use this to unset). - cookbook ran by andrew@buster
* 14:44 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:44 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:43 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: b46234ee-9900-4708-a815-{{Gerrit|4324379be48c}}, use this to unset). - cookbook ran by andrew@buster
* 14:43 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:43 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:42 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:39 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:39 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:38 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:38 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 12bc87a9-7602-4795-818c-{{Gerrit|f7151b1c9ead}}, use this to unset). - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:35 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: ba787a26-62db-4662-ade2-{{Gerrit|2e3061b7390d}}, use this to unset). - cookbook ran by andrew@buster
* 14:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:29 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 6f0e79c6-2d91-478e-92df-{{Gerrit|0f65de48212f}}, use this to unset). - cookbook ran by andrew@buster
* 14:29 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 7f577c0f-0182-498f-b198-{{Gerrit|307d51df8c3a}}, use this to unset). - cookbook ran by andrew@buster
* 14:28 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: bd1bff92-bfb8-483d-a719-{{Gerrit|13fbdf3d8b21}}, use this to unset). - cookbook ran by andrew@buster
* 14:27 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:27 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: c9cd7dbb-c5ad-4364-a8e0-{{Gerrit|afb52ef98683}}, use this to unset). - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 4deca8a4-f73f-4b42-b4be-{{Gerrit|acfe61421ed0}}, use this to unset). - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 6d0d58b2-8450-480d-94f2-{{Gerrit|93a257f25575}}, use this to unset). - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:13 dcaro: deleting all the leftover fullstack images (was due to max number of mysql connections reached)
* 04:51 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:47 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:46 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 432747a2-9a15-44d4-ba8a-{{Gerrit|4e7e87eb8b76}}, use this to unset). - cookbook ran by andrew@buster
* 04:45 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1c5adb8c-efcf-4036-bf22-{{Gerrit|0eef364ae399}}, use this to unset). - cookbook ran by andrew@buster
* 04:44 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:44 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:41 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 26feb897-9eb0-40b6-9525-{{Gerrit|4cf44f81638d}}, use this to unset). - cookbook ran by andrew@buster
* 04:41 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:40 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:40 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:39 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: cd2424d3-c9c9-4837-93b1-{{Gerrit|a6d8271e7873}}, use this to unset). - cookbook ran by andrew@buster
* 04:39 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:39 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1e85ec01-81c4-4664-b14f-{{Gerrit|b1cf6417988c}}, use this to unset). - cookbook ran by andrew@buster
* 04:38 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f498aaa4-fad3-42f8-92f4-{{Gerrit|48a98c108456}}, use this to unset). - cookbook ran by andrew@buster
* 04:37 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 0ed281f3-d9b2-4c0a-bf88-{{Gerrit|db42fe848b2f}}, use this to unset). - cookbook ran by andrew@buster
* 04:35 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f417dffc-fa1b-4e65-938a-{{Gerrit|49e54b6fac6d}}, use this to unset). - cookbook ran by andrew@buster
* 04:34 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:33 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 923926e0-7194-4e13-98c4-{{Gerrit|a4f703b2907b}}, use this to unset). - cookbook ran by andrew@buster
* 04:32 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:32 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:29 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: ca8f729b-92ad-45ec-b5da-{{Gerrit|55987b0fc9c2}}, use this to unset). - cookbook ran by andrew@buster
* 04:29 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:29 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: abe6992d-dfba-4e50-993d-{{Gerrit|de0a551a177a}}, use this to unset). - cookbook ran by andrew@buster
* 04:24 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:24 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: df91107d-bf34-49db-9756-{{Gerrit|44ad06162683}}, use this to unset). - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f45ec0b1-83c9-49e0-a3a8-{{Gerrit|897d5a48414f}}, use this to unset). - cookbook ran by andrew@buster
* 04:23 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:18 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 504ae503-bfad-41bd-9079-{{Gerrit|fb53a9c82a62}}, use this to unset). - cookbook ran by andrew@buster
* 04:17 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:17 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 8410faf9-667d-443d-bd17-{{Gerrit|bb3ca8e7f725}}, use this to unset). - cookbook ran by andrew@buster
* 04:15 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:11 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 40441a37-0dd7-44a8-960b-{{Gerrit|f74f98f53618}}, use this to unset). - cookbook ran by andrew@buster
* 04:10 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:10 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:08 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 517cff2b-a643-4714-a31f-{{Gerrit|6bbe0767656a}}, use this to unset). - cookbook ran by andrew@buster
* 04:08 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 50b63ad6-4d8b-4fcf-9c17-{{Gerrit|27a106b6ec52}}, use this to unset). - cookbook ran by andrew@buster
* 04:07 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: d912bdf5-54c7-490a-bdbe-{{Gerrit|ba9a23f07f4b}}, use this to unset). - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: e75f16d0-e42f-4d72-a3bd-{{Gerrit|be0cbc5f200a}}, use this to unset). - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 87f6c417-3703-4f73-9876-{{Gerrit|20f6767dc8e1}}, use this to unset). - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: be14f869-fdb0-4c38-84ff-{{Gerrit|3d1e00d227eb}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: feb93ea7-2c64-4831-b140-{{Gerrit|dad8311dbcef}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: c65a23e3-a809-4a75-88e4-{{Gerrit|29a8b351ecdb}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:00 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:00 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:57 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:56 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8c8f64ca-a4e8-47cf-a2ce-{{Gerrit|23059109fb6a}}, use this to unset). - cookbook ran by andrew@buster
* 03:55 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:55 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:54 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 9d0457e8-a861-46ce-ac35-{{Gerrit|53975a46018e}}, use this to unset). - cookbook ran by andrew@buster
* 03:53 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:53 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:40 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:38 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:30 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 851d3b34-68f7-4c57-920b-{{Gerrit|2a64a9ea573d}}, use this to unset). - cookbook ran by andrew@buster
* 03:29 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:29 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:17 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 1060ab82-d53a-4d48-8e15-{{Gerrit|3198cabcfc2f}}, use this to unset). - cookbook ran by andrew@buster
* 03:17 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:17 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: dde1dc09-44b5-42af-ac49-{{Gerrit|c58dbbae7cb4}}, use this to unset). - cookbook ran by andrew@buster
* 03:14 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: b59fa99b-8713-4e63-8c56-{{Gerrit|08bd71fdfbe3}}, use this to unset). - cookbook ran by andrew@buster
* 03:13 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:07 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 3547c843-6b15-4151-b452-{{Gerrit|6f618a886b7b}}, use this to unset). - cookbook ran by andrew@buster
* 03:07 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 6f5808d0-afb2-4222-ae8e-{{Gerrit|4d953d49cd63}}, use this to unset). - cookbook ran by andrew@buster
* 03:06 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: da738ba3-883f-4d76-bb33-{{Gerrit|7eb551de5fc2}}, use this to unset). - cookbook ran by andrew@buster
* 03:05 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:05 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:04 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:04 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:03 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f90ca4c1-78af-46bd-9262-{{Gerrit|0fa4edc2898e}}, use this to unset). - cookbook ran by andrew@buster
* 03:02 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:00 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:00 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 62383663-9ae6-4a15-a2a6-{{Gerrit|62be06adbc96}}, use this to unset). - cookbook ran by andrew@buster
* 02:59 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:59 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:59 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 6e0da344-49ad-451f-8b8d-{{Gerrit|ea8f853a6f89}}, use this to unset). - cookbook ran by andrew@buster
* 02:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:52 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: c9ec5b7c-aa9f-4afd-a417-{{Gerrit|54c52c07ca47}}, use this to unset). - cookbook ran by andrew@buster
* 02:51 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:51 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:48 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 5509c648-58f6-49fd-90e8-{{Gerrit|ac9aa66e8b04}}, use this to unset). - cookbook ran by andrew@buster
* 02:48 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:48 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: af4ba9ad-e765-4087-86aa-{{Gerrit|9111c0821175}}, use this to unset). - cookbook ran by andrew@buster
* 02:46 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:45 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:45 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: 6da98123-8838-4d2e-8659-{{Gerrit|d769bfa292fe}}, use this to unset). - cookbook ran by andrew@buster
* 02:44 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:44 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:43 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 02:43 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:42 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:42 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 91860079-8fff-40db-9de7-{{Gerrit|17741d528ad6}}, use this to unset). - cookbook ran by andrew@buster
* 02:41 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:41 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:39 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: a6a0f414-d305-4a09-87d2-{{Gerrit|066eec83642b}}, use this to unset). - cookbook ran by andrew@buster
* 02:39 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 6000898c-4e5e-49c8-8841-{{Gerrit|3262d6da0366}}, use this to unset). - cookbook ran by andrew@buster
* 02:37 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:37 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 24a8b50e-5c04-40e6-a732-{{Gerrit|f0232b83e240}}, use this to unset). - cookbook ran by andrew@buster
* 02:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8d259aca-1803-4346-b5a4-{{Gerrit|a447673b3537}}, use this to unset). - cookbook ran by andrew@buster
* 02:34 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:33 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 704aca7f-0069-4a4a-b8de-{{Gerrit|1aba7b9ca7ef}}, use this to unset). - cookbook ran by andrew@buster
* 02:32 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04c0fba3-e2cb-40b4-abfd-{{Gerrit|d24165c9bd55}}, use this to unset). - cookbook ran by andrew@buster
* 02:32 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 25e5e0c8-5c27-460e-9b44-{{Gerrit|1cbd1c2fb551}}, use this to unset). - cookbook ran by andrew@buster
* 02:28 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:25 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5e62968a-0bb6-43b1-ae5e-{{Gerrit|e352442ef976}}, use this to unset). - cookbook ran by andrew@buster
* 02:24 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:24 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:21 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5b1a8e8f-8699-433d-b764-{{Gerrit|197862dc5cf1}}, use this to unset). - cookbook ran by andrew@buster
* 02:20 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:20 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:15 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:15 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:13 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 299c466e-602b-4a8e-bb37-{{Gerrit|156553dbe426}}, use this to unset). - cookbook ran by andrew@buster
* 02:13 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:13 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:12 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:12 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:03 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6a64d619-c484-411f-ad9d-{{Gerrit|89d43b063737}}, use this to unset). - cookbook ran by andrew@buster
* 02:02 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:02 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 301309a2-a8b5-4698-98d6-{{Gerrit|bb4aa0a75e45}}, use this to unset). - cookbook ran by andrew@buster
* 00:46 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: b2f8c6a2-7224-4fb1-8e12-{{Gerrit|dff6dda02380}}, use this to unset). - cookbook ran by andrew@buster
* 00:46 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:40 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:40 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:38 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: c0f62e0e-7865-4e77-a238-{{Gerrit|d707ceed53a8}}, use this to unset). - cookbook ran by andrew@buster
* 00:38 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:38 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:36 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 96a368d5-232f-4187-b0bb-{{Gerrit|5923a50fff71}}, use this to unset). - cookbook ran by andrew@buster
* 00:36 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:36 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:35 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 31b6bfcb-81cb-4d89-b977-{{Gerrit|190899ea2e64}}, use this to unset). - cookbook ran by andrew@buster
* 00:35 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:35 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:28 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 50f763d4-36ea-4241-8e71-{{Gerrit|cc6aa20ccc9e}}, use this to unset). - cookbook ran by andrew@buster
* 00:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: cd3d17af-ee15-4a9f-8750-{{Gerrit|66b97a46ea12}}, use this to unset). - cookbook ran by andrew@buster
* 00:27 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 9b075166-d424-42c0-8e06-{{Gerrit|f74a47abb798}}, use this to unset). - cookbook ran by andrew@buster
* 00:21 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-07-18 ===
* 23:11 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: e0aab64f-d911-4d7e-9a97-{{Gerrit|c59d9868ceea}}, use this to unset). - cookbook ran by andrew@buster
* 23:11 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 447d9c7a-a00e-41db-93ed-{{Gerrit|6d2bfc66e5df}}, use this to unset). - cookbook ran by andrew@buster
* 23:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:01 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 46c2ddf2-50c3-4824-ba72-{{Gerrit|d7fb3fa4c5e0}}, use this to unset). - cookbook ran by andrew@buster
* 23:01 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6facdf4f-7bb8-42cd-8ac8-{{Gerrit|b4412fc4cf70}}, use this to unset). - cookbook ran by andrew@buster
* 23:00 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:52 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 23190573-fc0d-4872-8343-{{Gerrit|e35b84132028}}, use this to unset). - cookbook ran by andrew@buster
* 22:51 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:51 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:51 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 990313c4-5fa6-4de7-a8bf-{{Gerrit|b361f0c79e90}}, use this to unset). - cookbook ran by andrew@buster
* 22:50 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:50 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:48 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:48 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:42 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 59367d39-bd2f-47fd-becd-{{Gerrit|255bf823ff12}}, use this to unset). - cookbook ran by andrew@buster
* 22:42 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:42 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:40 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 38602815-d187-4455-9676-{{Gerrit|59607505bba7}}, use this to unset). - cookbook ran by andrew@buster
* 22:39 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 00f21e1b-d014-4bc5-995f-{{Gerrit|e25201fc77c4}}, use this to unset). - cookbook ran by andrew@buster
* 22:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:31 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: d05d07b2-8c60-46de-a89e-{{Gerrit|76987901901b}}, use this to unset). - cookbook ran by andrew@buster
* 22:31 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 47f6b9e6-ceab-4fb6-8f76-{{Gerrit|0de4341d0841}}, use this to unset). - cookbook ran by andrew@buster
* 22:31 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:31 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 599042a5-da25-4985-b502-{{Gerrit|328fe40a72d5}}, use this to unset). - cookbook ran by andrew@buster
* 22:29 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:29 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:29 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: a554d107-c14d-41f8-b180-{{Gerrit|093e2cf73e72}}, use this to unset). - cookbook ran by andrew@buster
* 22:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9fef70c7-6f7d-4b33-a9bc-{{Gerrit|8c39630dc182}}, use this to unset). - cookbook ran by andrew@buster
* 22:26 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:02 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 3744c0fa-084d-4198-98ca-{{Gerrit|66c7a5210f41}}, use this to unset). - cookbook ran by andrew@buster
* 22:02 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 01caf1e1-df64-474a-94d5-{{Gerrit|f37751e23d62}}, use this to unset). - cookbook ran by andrew@buster
* 22:01 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:59 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 3018c2fc-9cd5-45c8-a160-{{Gerrit|d5ad5728c8d5}}, use this to unset). - cookbook ran by andrew@buster
* 21:59 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:59 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: d760f916-32ea-4194-8974-{{Gerrit|f36064a9866c}}, use this to unset). - cookbook ran by andrew@buster
* 21:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:00 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: cb2a38f3-e9ca-4bf6-989c-{{Gerrit|a27be2b2d88c}}, use this to unset). - cookbook ran by andrew@buster
* 20:59 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:58 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 93fa9477-54b8-46e7-8508-{{Gerrit|a2d492528d1f}}, use this to unset). - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f6cf5b29-ce8b-426e-af1c-{{Gerrit|cab74e0c4a3a}}, use this to unset). - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: fe26e612-d76e-4120-b60a-{{Gerrit|4f300add224f}}, use this to unset). - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 2abd94be-b6d8-4042-a9ce-{{Gerrit|998e41d932f4}}, use this to unset). - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 10973c71-30d8-44bf-a310-{{Gerrit|d0c1af76d399}}, use this to unset). - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f8f1b214-ec76-4157-a8e6-{{Gerrit|6374e5d14608}}, use this to unset). - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:11 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 95dae613-2b04-472e-a8f9-{{Gerrit|e71c3fbb8d0e}}, use this to unset). - cookbook ran by andrew@buster
* 20:10 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: c804f4db-ea63-44e9-aff8-{{Gerrit|fea54c238345}}, use this to unset). - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 597a901d-4624-43de-b986-{{Gerrit|c4c6c2fcf80c}}, use this to unset). - cookbook ran by andrew@buster
* 20:03 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:03 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: fdceed0d-6fa0-4806-8fb6-{{Gerrit|3b321a5a1623}}, use this to unset). - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:37 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: b0904345-9aa3-4202-b992-{{Gerrit|78644141517c}}, use this to unset). - cookbook ran by andrew@buster
* 19:36 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:31 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:31 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-07-13 ===
* 14:48 bd808: Added Tavvi as member of #acl*wmcs-team

=== 2022-07-12 ===
* 12:04 wm-bot2: Ceph cluster at &lt;nowiki&gt;{&lt;/nowiki&gt;self.deployment&lt;nowiki&gt;}&lt;/nowiki&gt; set out of maintenance. - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Set the ceph cluster for codfw1dev in maintenance, alert silence ids: db32805f-a033-4e0e-8fc7-{{Gerrit|ed0c2e9f8be1}},f4e698f0-4b51-4b07-9ba8-{{Gerrit|0b296ca1c4fb}},39ad5325-44ed-44d1-bba5-{{Gerrit|91a85c7a8401}},a584a3c6-9dae-41e9-ac82-{{Gerrit|a91cd35af8fb}} - cookbook ran by dcaro@vulcanus
* 08:37 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2005-dev', 'cloudnet2006-dev'] - cookbook ran by dcaro@vulcanus
* 08:37 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:31 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2005-dev,cloudnet2006-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-08 ===
* 15:57 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-07-07 ===
* 07:17 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T312509|T312509]]) - cookbook ran by dcaro@vulcanus
* 07:12 wm-bot2: Rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T312509|T312509]]) - cookbook ran by dcaro@vulcanus

=== 2022-07-06 ===
* 17:50 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: ['8a5b9eee-48c0-474d-8277-faeb05a2ea61', '65aad0fc-d887-47a3-b20c-d1ed461a2411', '86b078ae-3a27-4063-8c7c-198a2fe0c172'] - cookbook ran by dcaro@vulcanus

=== 2022-07-04 ===
* 13:27 wm-bot2: Rebooting cloudgw host cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot2: Rebooted cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Rebooting cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Rebooting all the cloudgw nodes from the eqiad1 deployment: cloudgw1001.eqiad.wmnet,cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:13 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 13:12 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:09 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:56 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:52 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:56 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 10:56 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:51 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:51 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:47 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:47 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:41 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 09:07 wm-bot2: Rebooting cloudnet host cloudnet1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:06 wm-bot2: Rebooted cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot2: Rebooting cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot2: Rebooting all the cloudnet nodes cloudnet1003,cloudnet1004 - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:53 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:53 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:47 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:47 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:55 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:44 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 07:44 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:36 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:36 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:09 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:05 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:04 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-03 ===
* 21:27 andrewbogott: rebuilding rabbit cluster in codfw1dev to get rid of some queues so unresponsive that they can't otherwise be deleted

=== 2022-07-02 ===
* 11:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-01 ===
* 15:35 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:30 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:29 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:25 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:42 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 14:42 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:32 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:02 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:58 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:58 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:54 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:52 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:37 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:34 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:30 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-06-30 ===
* 18:17 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 18:13 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:45 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 06:52 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-06-29 ===
* 08:45 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 08:41 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-06-28 ===
* 13:03 taavi: grant the tools project access to the g3.cores16.ram64.disk20.10xiops flavor [[phab:T301949|T301949]]

=== 2022-06-22 ===
* 16:48 taavi: restart designate-*.service on both cloudservices nodes
* 16:44 taavi: restart nova-conductor on all the cloudcontrol nodes
* 12:50 andrewbogott: rebooting each eqiad1 cloudcontrol node in hopes of getting a baseline re: openstack instability

=== 2022-06-21 ===
* 04:48 andrewbogott: stopping nova-fullstack agent on cloudcontrol1003; it's going to page us otherwise and we're all AFK tomorrow
* 04:02 andrewbogott: restarting rabbitmq on cloudcontrol100x (one at a time)

=== 2022-06-17 ===
* 17:53 andrewbogott: switching to a new python-based health check for galera and haproxy. This may make things more stable, or it may not. [[phab:T310664|T310664]]
* 06:15 taavi: restart neutron-linuxbridge-agent on cloudvirt1046

=== 2022-06-15 ===
* 11:34 taavi: restart neutron-linuxbridge-agent on cloudvirt1022

=== 2022-06-14 ===
* 16:26 wm-bot2: OSDs (['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloudcephosd1021', 'cl
* 14:38 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudceph
* 12:55 wm-bot2: OSDs (['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 12:44 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus

=== 2022-06-13 ===
* 11:14 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:07 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:05 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:04 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:03 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 09:15 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus

=== 2022-06-06 ===
* 13:21 andrewbogott: restarting mysql/galera on cloudcontrol100x in an attempt to stabilize some flapping there

=== 2022-06-02 ===
* 07:51 taavi: restart neutron-linuxbridge-agent.service on cloudvirt1034 [[phab:T309732|T309732]]
* 00:37 andrewbogott: updated nameservers for codfw1dev instances via 'openstack subnet set --dns-nameserver etc.'

=== 2022-06-01 ===
* 17:11 andrewbogott: restarting designate services in cloudservices1xxx hosts
* 16:37 taavi: root@cloudcontrol1005:~# cinder reset-state --state available 491066a4-16a9-4ce4-a9f6-{{Gerrit|182660616d77}} for [[phab:T309659|T309659]]

=== 2022-05-30 ===
* 17:43 andrewbogott: restarting neutron-rpc and neutron-api services on cloudcontrol1xxx

=== 2022-05-29 ===
* 14:55 andrewbogott: restarting nova services on all eqiad1 cloudcontrol nodes to recover from rabbit breakage
* 14:15 andrewbogott: restarting rabbitmq on all eqiad1 cloudcontrol nodes (one at a time)

=== 2022-05-25 ===
* 20:01 balloons: clean up cinder backup a bit, restart service due to network outage
* 20:01 balloons: cloudvirt1029 restarted nova due to network outage

=== 2022-05-19 ===
* 15:21 andrewbogott: resetting password for the 'troveguest' rabbitmq user. I think I may have broken this during a recent rebuild of the rabbitmq cluster

=== 2022-05-18 ===
* 15:42 andrewbogott: updated the 'debian-11.0-bullseye' glance image with a fresh build

=== 2022-05-14 ===
* 11:33 taavi: deleted projects 'ores' and 'ores-staging' [[phab:T308102|T308102]]

=== 2022-05-13 ===
* 06:20 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 06:20 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:16 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:16 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:15 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:15 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:11 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 06:11 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:10 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:10 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:09 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:07 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:06 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:06 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:05 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:51 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:50 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:50 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:47 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:47 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:44 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:44 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:41 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:40 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:40 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:38 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:37 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:37 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:30 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:29 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:29 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:19 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:12 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:12 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:08 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:01 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:52 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:48 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:48 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:44 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:31 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:24 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:21 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:21 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:18 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:37 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:36 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:36 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:34 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:34 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:27 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:27 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:25 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:25 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:55 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:55 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:04 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:23 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 01:23 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:19 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:01 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:01 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:25 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 00:25 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:20 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:19 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:19 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:11 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 00:11 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:07 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-12 ===
* 23:55 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:55 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:23 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:23 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:20 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:17 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:17 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:13 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:57 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:56 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:56 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:55 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:55 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:54 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:54 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:53 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:52 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:52 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:51 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:44 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:44 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:42 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:36 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:36 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:33 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:33 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:30 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:22 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:22 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:21 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:21 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:11 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:11 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:08 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:08 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:07 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:07 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:05 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:04 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:04 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:00 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:55 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:55 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:53 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:50 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:48 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:48 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:44 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:44 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:43 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:43 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:31 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:23 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:22 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:22 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:11 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:07 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:00 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:58 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:35 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:35 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:06 andrewbogott: stopping nfs-server on labstore1004 in preparation for reboot
* 04:12 andrewbogott: rebooting primary bastion (bastion-eqiad1-03.bastion.eqiad1.wikimedia.cloud) in hopes of resolving a problem with ssh proxying

=== 2022-05-11 ===
* 18:48 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:39 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:38 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:38 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:04 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:03 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:03 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 08:56 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 08:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 07:53 dcaro: test
* 04:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:43 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:43 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:39 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:23 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:09 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:08 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:08 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:04 andrewbogott: reset and recreated the rabbitmq cluster in eqiad1 to get around some broken queues.
* 03:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:25 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:25 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-10 ===
* 21:43 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:40 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:40 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:35 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:32 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:05 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:02 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:55 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:55 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:47 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:46 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:45 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:45 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:40 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:39 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:39 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:37 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:36 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:33 wm-bot: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 19:33 wm-bot: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:29 wm-bot: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:06 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:05 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:05 wm-bot: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:41 andrewbogott: rebooting cloud*-dev for [[phab:T307668|T307668]]
* 13:59 taavi: manually attached [[User:Dreamy Jazz]] to wikitech for a password reset (https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Manually_associate_an_LDAP_account_with_wikitech)

=== 2022-05-07 ===
* 01:33 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:32 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:30 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:21 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:18 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-03 ===
* 20:38 andrewbogott: upgrading clouddb2001-dev in place
* 18:18 taavi: updated 'puppet-enc' endpoints on the keystone catalog to use https and port 443

=== 2022-05-02 ===
* 16:56 dcaro: rebooting cloudmetrics1001

=== 2022-04-29 ===
* 14:22 andrewbogott: changing login.toolforge.org, bastion.toolforge.org, and dev.toolforge.org dns entries to refer to the new Buster bastions [[phab:T277653|T277653]] https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Timeline

=== 2022-04-27 ===
* 14:51 wm-bot: Finished rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloud
* 14:50 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:41 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:41 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot: Finished rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot: Rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot: Finished rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:01 wm-bot: Rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:01 wm-bot: Finished rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Finished rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:44 wm-bot: Rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:43 wm-bot: Finished rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:39 wm-bot: Rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:39 wm-bot: Finished rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:35 wm-bot: Rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:35 wm-bot: Finished rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Finished rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:26 wm-bot: Rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:26 wm-bot: Finished rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot: Rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot: Finished rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot: Rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot: Finished rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot: Rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot: Finished rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Finished rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:57 wm-bot: Rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:57 wm-bot: Finished rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Finished rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Finished rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:46 wm-bot: Rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:46 wm-bot: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024 - cookbo
* 12:15 wm-bot: Finished rebooting the nodes ['cloudcephmon1001', 'cloudcephmon1002', 'cloudcephmon1003'] - cookbook ran by dcaro@vulcanus
* 12:15 wm-bot: Finished rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:12 wm-bot: Rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:12 wm-bot: Finished rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:09 wm-bot: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:09 wm-bot: Finished rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:07 wm-bot: Rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:07 wm-bot: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:02 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:02 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:59 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:59 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 11:55 wm-bot: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] - cookbook ran by dcaro@vulcanus
* 11:55 wm-bot: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot: Rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:47 wm-bot: Rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:47 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:43 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:43 wm-bot: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev - cookbook ran by dcaro@vulcanus

=== 2022-04-26 ===
* 10:36 taavi: [codfw1dev] updated designate pool to 2004/2005-dev according to the instructions on https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS/Designate#Initial_designate/pdns_node_setup

=== 2022-04-22 ===
* 10:33 taavi: [codfw1dev] restart designate-sink on both new cloudservices host to fix rabbitmq connectivity

=== 2022-04-21 ===
* 05:38 andrewbogott: replaced cloudservices200[2,3] with cloudservices200[4,5]

=== 2022-04-19 ===
* 15:29 andrewbogott: stopping all VMs on cloudvirt1019, reimaging host

=== 2022-04-18 ===
* 15:23 andrewbogott: reimaging cloudvirt1020, leaving VMs in place
* 13:40 andrewbogott: shutting down many codfdfw1dev servers (including network infra!) for [[phab:T305469|T305469]]

=== 2022-04-14 ===
* 20:14 andrewbogott: restarting nova-api and nova-conductor services in a superstitious attempt to reduce open DB connections

=== 2022-04-13 ===
* 22:01 andrewbogott: restarting galera on cloudcontrols (one by one) to clear open connections

=== 2022-04-11 ===
* 15:59 taavi: created cloudinfra.wmcloud.org zone

=== 2022-04-09 ===
* 19:55 andrewbogott: reimaging cloudbackup1001-dev to bullseye
* 19:37 taavi: add 'puppet-enc' service &amp; endpoint to keystone [[phab:T274666|T274666]]
* 19:25 andrewbogott: reimaging cloudbackup1002-dev to bullseye

=== 2022-04-07 ===
* 12:51 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T305631|T305631]]) - cookbook ran by arturo@nostromo

=== 2022-04-06 ===
* 09:12 arturo: [codf1dev] installing python3-eventlet 0.30.2-5~bpo11+1 on all required servers (cloudvirt, cloudnet, cloudcontrol) ([[phab:T305157|T305157]])
* 08:45 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudvirt2003-dev ([[phab:T305157|T305157]])
* 08:42 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudcontrol servers ([[phab:T305157|T305157]])
* 08:24 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudvirt2003-dev ([[phab:T305157|T305157]])
* 08:20 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudcontrol servers ([[phab:T305157|T305157]])

=== 2022-03-30 ===
* 11:20 arturo: apply urpf strict filter to eqiad cloud-hosts vlan - [[phab:T285461|T285461]]

=== 2022-03-29 ===
* 10:02 dcaro: restarting keystone ([[phab:T304918|T304918]])

=== 2022-03-23 ===
* 22:53 wm-bot: Drained 'cloudvirt1045.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:38 wm-bot: Drained 'cloudvirt1044.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:12 wm-bot: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:12 wm-bot: Draining 'cloudvirt1045.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:08 wm-bot: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:07 wm-bot: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:06 wm-bot: Draining 'cloudvirt1044.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:06 wm-bot: Draining 'cloudvirt1043.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:54 wm-bot: Drained 'cloudvirt1042.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:19 wm-bot: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:19 wm-bot: Draining 'cloudvirt1042.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:12 wm-bot: Drained 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:12 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:09 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:07 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:04 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:55 wm-bot: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:54 wm-bot: Draining 'cloudvirt1041.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:30 wm-bot: Drained 'cloudvirt1039.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:14 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:14 wm-bot: Draining 'cloudvirt1039.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:44 wm-bot: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:43 wm-bot: Draining 'cloudvirt1038.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:19 wm-bot: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:18 wm-bot: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:13 wm-bot: Drained 'cloudvirt1036.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:02 wm-bot2: Testing wm-bot relay to #wikimedia-cloud-feed
* 17:55 wm-bot: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:54 wm-bot: Draining 'cloudvirt1036.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:04 wm-bot: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:03 wm-bot: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:03 wm-bot: Drained 'cloudvirt1034.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:51 wm-bot: Drained 'cloudvirt1033.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:37 wm-bot: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:37 wm-bot: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Draining 'cloudvirt1034.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 15:01 wm-bot: Drained 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 15:00 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:57 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:44 wm-bot: Drained 'cloudvirt1031.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:35 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:34 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:32 wm-bot: Drained 'cloudvirt1030.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:20 wm-bot: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:19 wm-bot: Draining 'cloudvirt1031.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:18 wm-bot: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:17 wm-bot: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:54 taavi: restart nova-fullstack on cloudcontrol1003 to pick up bastion ip change
* 13:43 wm-bot: Drained 'cloudvirt1029.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:23 wm-bot: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:22 wm-bot: Draining 'cloudvirt1029.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-22 ===
* 22:59 wm-bot: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:58 wm-bot: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-17 ===
* 01:09 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:53 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:52 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:52 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-15 ===
* 20:58 wm-bot: Drained 'cloudvirt1026.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:14 wm-bot: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 13:14 wm-bot: Unsetting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 10:32 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 10:30 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo

=== 2022-03-14 ===
* 21:24 wm-bot: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:59 wm-bot: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:58 wm-bot: Setting cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:58 wm-bot: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:02 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:59 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:59 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:16 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:13 wm-bot: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:56 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:55 wm-bot: Setting cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:55 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:53 wm-bot: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:52 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:51 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:51 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:50 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 11:48 dcaro: rebased cookbooks on latest master, make sure you pull before sending new patches

=== 2022-03-08 ===
* 18:29 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:29 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:29 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:23 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:21 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:21 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:18 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:17 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:17 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:27 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:27 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:18 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:15 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:15 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:47 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:47 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:08 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:07 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:07 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:11 arturo: [codfw1dev] rebooting cloudservices servers for [[phab:T303179|T303179]]
* 13:07 arturo: [codfw1dev] rebooting cloudvirt servers for [[phab:T303179|T303179]]
* 13:06 arturo: [codfw1dev] rebooting cloudnet servers for [[phab:T303179|T303179]]
* 12:55 arturo: [codfw1dev] rebooting cloudcontrol servers for [[phab:T303179|T303179]]

=== 2022-03-03 ===
* 08:49 taavi: deploying cloudmetrics grafana to grafana 8, [[phab:T282863|T282863]]

=== 2022-03-02 ===
* 09:06 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/701347

=== 2022-02-28 ===
* 15:30 dcaro: cleaning up leftover snapshots from failed backups of the maps volume ([[phab:T302720|T302720]])

=== 2022-02-24 ===
* 17:04 andrewbogott: upgrading eqiad1 and codfw1dev to mariadb 10.5.15+maria~bullseye via 'apt-get install libmariadb3:amd64 galera-4 mariadb-server'
* 15:42 dcaro: stopping and starting mariadb on cloudcontrol1003 ([[phab:T302146|T302146]])
* 10:37 arturo: [codfw1dev] briefly installed galera-4 (26.4.11+1bullseye) over (26.4.9-0+deb11u1) on cloudcontrol2001-dev and then downgrade again to verify package install ([[phab:T302482|T302482]])

=== 2022-02-23 ===
* 20:39 taavi: added domain-wide 'designateadmin' and 'observer' roles to project-proxy-dns-manager service account [[phab:T295246|T295246]]
* 17:40 andrewbogott: restarting lots of openstack services to try to clear up the mess that is [[phab:T236101|T236101]]
* 12:13 arturo: cleaning up cinder volume snapshots, aborrero@cloudcontrol1005:~$ for i in $(sudo wmcs-openstack volume snapshot list -f value -c ID) ; do sudo wmcs-openstack volume snapshot delete $i ; done ([[phab:T302382|T302382]])
* 10:14 arturo: cleaning up neutron agents for non-existent servers cloudvirt100[1-9].eqiad.wmnet,cloudvirt10[12-15].eqiad.wmnet
* 10:05 dcaro: Deleting stuck novafullstack servers, to let the service create new ones ([[phab:T302369|T302369]])
* 09:56 arturo: neutron agent-delete bad663b3-fd25-4393-a546-{{Gerrit|4b1b4bdec4db}} (Linux bridge agent {{!}} cloudvirtan1001)
* 09:56 arturo: neutron agent-delete 1071c198-ed57-4b5a-9439-{{Gerrit|30e66a31aa69}} (Linux bridge agent {{!}} cloudvirtan1005)
* 09:55 arturo: neutron agent-delete 2eeef198-8af7-4e5d-bd73-{{Gerrit|e14a2a8d2404}} (Linux bridge agent {{!}} cloudvirtan1004)
* 09:55 arturo: neutron agent-delete afe173eb-35ba-444a-9960-{{Gerrit|899629786d2f}} (Linux bridge agent {{!}} cloudvirtan1003)
* 09:54 arturo: neutron agent-delete afcb9b7f-c1a6-4ff4-9b10-{{Gerrit|92bfbe8d1a56}} (Linux bridge agent {{!}} cloudvirtan1002)
* 09:39 dcaro: restarting neutron-api cloudcontrol1003 to see if the agent status update starts working ([[phab:T302369|T302369]])
* 09:38 dcaro: restarting neutron-dhcp-agent on cloudnet1003 ([[phab:T302369|T302369]])

=== 2022-02-22 ===
* 22:10 andrewbogott: raising project 'maps' quota by two tb -- [[phab:T300160|T300160]]
* 09:24 arturo: restarting mariadb @ cloudcontrol1003 ([[phab:T302146|T302146]])
* 09:13 arturo: restarting mariadb @ cloudcontrol1004 ([[phab:T302146|T302146]])

=== 2022-02-18 ===
* 21:57 andrewbogott: leaving cloudcontrol1003 downtimed with disabled puppet for the weekend.  Everything there should be stable and fine save rabbit which needs an upgrade.
* 21:30 andrewbogott: rebooting cloudcontrol1003 because rabbit is freaking out
* 17:25 andrewbogott: in-place upgrade of cloudcontrol1004 to bullseye -- [[phab:T281276|T281276]]
* 12:34 arturo: manually install prometheus-openstack-exporter on cloudcontrol1005 ([[phab:T302050|T302050]])

=== 2022-02-17 ===
* 23:02 andrewbogott: in-place upgrade to Bullseye on cloudcontrol1005 [[phab:T281276|T281276]]

=== 2022-02-15 ===
* 14:15 taavi: [codfw1dev] added domain-wide 'designateadmin' and 'observer' roles to codfw1dev-proxy-dns-manager service account [[phab:T295246|T295246]]

=== 2022-02-04 ===
* 10:12 arturo: restart backup_vms service in cloudvirt1024 ([[phab:T300956|T300956]])

=== 2022-02-03 ===
* 08:21 taavi: cloudmetrics1004: manually added an empty line to /etc/prometheus/blackbox.yml to make /usr/local/bin/blackbox-exporter-assemble happy (clearing &quot;performing a change every puppet run&quot; alert)

=== 2022-02-02 ===
* 02:36 andrewbogott: restarting mariadb on cloudcontrol1004

=== 2022-01-31 ===
* 10:15 arturo: cloudcontrol1005:~$ sudo systemctl restart backup_glance_images.service (failed state, no logs, icinga alert)

=== 2022-01-29 ===
* 18:24 taavi: delete 2 puppet prefixes in a weird state [[phab:T299750|T299750]]

=== 2022-01-27 ===
* 13:24 arturo: cloudmetrics1004:~ $ sudo systemctl restart wmcs_monitoring_graphite_rsync.service ([[phab:T300138|T300138]])

=== 2022-01-26 ===
* 19:09 andrewbogott: bootstrapping a fresh galera node on cloudcontrol1004
* 18:57 andrewbogott: restarting mariadb on cloudcontrol1004

=== 2022-01-25 ===
* 10:49 arturo: made cloudmetrics1001/1002 primary/backup respectively ([[phab:T299744|T299744]], [[phab:T297814|T297814]], [[phab:T300011|T300011]])

=== 2022-01-19 ===
* 16:38 andrewbogott: moving all scratch mounts to scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud

=== 2022-01-05 ===
* 03:11 andrewbogott: 'cp /etc/apt/sources.list /etc/apt/sources.list.prepuppet' on all VMs. Backing up state before puppetizing sources.list with https://gerrit.wikimedia.org/r/c/operations/puppet/+/751498

=== 2022-01-04 ===
* 12:44 dcaro: increasing the size_limit for labs ldap servers

=== 2021-12-26 ===
* 16:55 majavah: run attachLdapUser.php on wikitech for developer account &quot;Karthiksripal&quot;

=== 2021-12-24 ===
* 22:51 majavah: ran the wikireplica dns script on s5 [[phab:T298303|T298303]]

=== 2021-12-23 ===
* 21:42 majavah: deployed horizon wmf-proxy-dashboard update to fix editing of existing proxies

=== 2021-12-21 ===
* 10:39 arturo: dropped egress NAT exceptions for WMF apt repos, [[phab:T298042|T298042]]

=== 2021-12-15 ===
* 12:44 dcaro: Downtiming cloudvirt-wdqs1001 as it has no VMs running until disk space is fixed ([[phab:T297454|T297454]])

=== 2021-12-14 ===
* 10:26 dcaro: Moved the nova cache (/var/lib/nova/instances/_base) and the canary image local data (/var/lib/nova/instance/&lt;canary_image_id&gt;) to the root disk on cloudvirt-wdqs1001 to temporary free some space ([[phab:T297454|T297454]])

=== 2021-12-13 ===
* 18:08 wm-bot: Drained 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:50 wm-bot: Set cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:49 wm-bot: Setting cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:49 wm-bot: Draining 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:44 wm-bot: Drained 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:30 wm-bot: Set cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:30 wm-bot: Setting cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:30 wm-bot: Draining 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:13 wm-bot: Drained 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
* 16:50 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:47 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:47 wm-bot: Draining 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
* 16:44 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:43 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse

=== 2021-12-03 ===
* 18:56 andrewbogott: maintain-views and maintain-meta-p on clouddb1013-1020
* 10:49 majavah: deleting dbbackups-dashboard project [[phab:T296992|T296992]]

=== 2021-12-02 ===
* 01:17 wm-bot: Drained 'cloudvirt1028.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:56 wm-bot: Set cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:56 wm-bot: Setting cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:56 wm-bot: Draining 'cloudvirt1028.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:50 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:50 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:50 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:28 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:03 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:02 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:02 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster

=== 2021-12-01 ===
* 23:59 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:59 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 23:54 andrewbogott: *correction* adding spare cloudvirts 1044 and 1045 to the 'ceph' pool in order to make space for future juggling around [[phab:T296790|T296790]] and [[phab:T296792|T296792]]
* 23:53 andrewbogott: adding spare cloudvirts 1044 and 1055 to the 'ceph' pool in order to make space for future juggling around [[phab:T296790|T296790]] and [[phab:T296792|T296792]]

=== 2021-11-28 ===
* 17:48 andrewbogott: moved cloudvirt1018 out of the 'localstorage' aggregate and into 'maintenance' for [[phab:T296592|T296592]].  It will need to be moved back after the raid is rebuilt.

=== 2021-11-21 ===
* 07:19 dcaro_away: restarting designate-sink with some extra logs in it ([[phab:T296144|T296144]])

=== 2021-11-17 ===
* 15:48 andrewbogott: upgrading mariadb packages on eqiad1 cloudcontrols
* 15:39 andrewbogott: sudo cumin  &quot;cloud*&quot; 'apt-get update -y --allow-releaseinfo-change'
* 15:26 andrewbogott: updated mariadb packages on codfw1dev cloudcontrols to 1:10.3.31-0+deb10u1

=== 2021-11-12 ===
* 13:31 arturo: restarting glance-api services to make sure they work with new ceph auth creds ([[phab:T293752|T293752]])

=== 2021-11-08 ===
* 21:50 andrewbogott: returned clouddb pools back to normal after maintain_views run: https://gerrit.wikimedia.org/r/c/operations/puppet/+/737505 [[phab:T216481|T216481]]
* 20:07 andrewbogott: depooling clouddb1013 for maintain_views attempt
* 10:54 arturo: [codfw1dev] create service account `srv-networktests` following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for [[phab:T294955|T294955]]
* 10:34 arturo: create service account `srv-networktests` following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for [[phab:T294955|T294955]]

=== 2021-11-05 ===
* 11:18 wm-bot: Added 1 new OSDs ['cloudcephosd1024.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:17 wm-bot: Added OSD cloudcephosd1024.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:15 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:12 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:12 wm-bot: Adding OSD cloudcephosd1024.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:12 wm-bot: Adding new OSDs ['cloudcephosd1024.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance

=== 2021-11-04 ===
* 16:39 wm-bot: Added 1 new OSDs ['cloudcephosd1023.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:39 wm-bot: Added OSD cloudcephosd1023.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:37 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:34 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:33 wm-bot: Adding OSD cloudcephosd1023.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:33 wm-bot: Adding new OSDs ['cloudcephosd1023.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:17 wm-bot: Added 1 new OSDs ['cloudcephosd1022.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:17 wm-bot: Added OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:16 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:13 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:12 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:12 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:00 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:00 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:26 wm-bot: Added 1 new OSDs ['cloudcephosd1021.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:26 wm-bot: Added OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:23 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:20 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:19 wm-bot: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:19 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:16 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance

=== 2021-11-03 ===
* 17:22 arturo: [codfw1dev] installing keepalived 2.1.5 from buster-backports on cloudgw2001-dev/2002-dev ([[phab:T294956|T294956]])
* 11:45 arturo: [codfw1dev] downgrade kernel on cloudgw2001-dev/2002-dev ([[phab:T294853|T294853]], [[phab:T291813|T291813]])

=== 2021-11-02 ===
* 10:54 arturo: rebooting cloudnet1004/1003 for [[phab:T291813|T291813]]
* 10:43 arturo: [codfw1dev] rebooting cloudgw200[12]-dev for [[phab:T291813|T291813]]

=== 2021-10-24 ===
* 00:47 andrewbogott: deploying a change so that openstack clients use tls endpoints: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732738

=== 2021-10-21 ===
* 10:19 arturo: drop firewall exception on core routers for wiki replicas legacy setup ([[phab:T293897|T293897]])
* 10:12 arturo: drop NAT exception for wiki replicas legacy setup ([[phab:T293897|T293897]])

=== 2021-10-20 ===
* 21:06 andrewbogott: creating cloudinfra-nfs project [[phab:T293936|T293936]]

=== 2021-10-18 ===
* 19:21 andrewbogott: also ticked the 'admin' box on wikitech for majavah [[phab:T292827|T292827]]
* 18:58 andrewbogott: granting majavah 'admin' role in the 'admin' project and also in the default domain.  [[phab:T292827|T292827]]

=== 2021-10-14 ===
* 12:28 arturo: [codfw1dev] add DB grants for cloudbackup2002.codfw.wmnet IP address to the cinder DB ([[phab:T292546|T292546]])

=== 2021-10-13 ===
* 10:46 arturo: updating python3-neutron across the fleet ([[phab:T292936|T292936]])

=== 2021-10-12 ===
* 09:06 dcaro: upgrading eqiad cloudnet hosts neutron packages ([[phab:T292936|T292936]])
* 08:57 dcaro: upgrading codfw cloudnet hosts neutron packages ([[phab:T292936|T292936]])

=== 2021-10-05 ===
* 09:39 arturo: [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) [[phab:T291257|T291257]]

=== 2021-09-30 ===
* 14:50 andrewbogott: sudo cumin  &quot;cloud*&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot; and sudo cumin  &quot;lab*&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot;  [[phab:T292202|T292202]]
* 14:43 andrewbogott: ran sudo cumin --force --timeout 500 -o json  &quot;A:all&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot; to get nslcd happy again [[phab:T292202|T292202]]

=== 2021-09-29 ===
* 09:41 arturo: [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured ([[phab:T291257|T291257]])

=== 2021-09-28 ===
* 16:23 bstorm: downtime for clouddb1020 to reduce re-pages in case this goes badly [[phab:T291963|T291963]]
* 16:21 bstorm: powering on clouddb1020 via remote console [[phab:T291963|T291963]]
* 15:58 bstorm: depooled clouddb1020 for repair [[phab:T291961|T291961]]
* 12:40 dcaro: Merged change on sssd for bullseye cloud hosts ([[phab:T291585|T291585]])
* 11:30 arturo: [codfw1dev] create floating IP 185.15.57.5 for manila-sharecontroller.cloudinfra-codfw1dev.codfw1dev.wmcloud.org ([[phab:T291257|T291257]])

=== 2021-09-27 ===
* 10:07 arturo: cloudcontrol1004 apparently healthy [[phab:T291446|T291446]]
* 09:25 arturo: rebooting cloudcontrol1004 for [[phab:T291446|T291446]]

=== 2021-09-24 ===
* 13:02 arturo: [codfw1dev] create VM manila-share-controller-01 on cloudinfra-codfw1dev
* 13:00 arturo: [codfw1dev] rebase labs/private.git on cloudinfra-puppetmaster-01, had merge conflict

=== 2021-09-21 ===
* 12:13 arturo: [codfw1dev] trying to create a manila service image ([[phab:T291257|T291257]])
* 11:45 arturo: [codfw1dev] created rabbitmq user ([[phab:T291257|T291257]])
* 11:32 arturo: [codfw1dev] populated manila DB &amp; created service endpoints ([[phab:T291257|T291257]])
* 11:06 arturo: [codfw1dev] give manila user admin role @ manila project ([[phab:T291257|T291257]])
* 11:06 arturo: [codfw1dev] created manila project ([[phab:T291257|T291257]])
* 10:57 arturo: [codfw1dev] created manila user @ labtestwikitech ([[phab:T291257|T291257]])
* 10:49 arturo: [codfw1dev] create manila database on cloudcontrol-dev nodes (galera) [[phab:T291257|T291257]]

=== 2021-09-20 ===
* 23:08 bstorm: ran `echo check &gt; /sys/block/md0/md/sync_action` on cloudcontrol1004 to check raid
* 22:48 andrewbogott: stopped puppet &amp; mariadb on cloudcontrol1004; it was flapping
* 22:44 andrewbogott: sudo touch /tmp/galera.disabled on cloudcontrol1004, the service seems troubled there
* 21:57 andrewbogott: moving cloudvirt1043 into the 'nfs' aggregate for [[phab:T291405|T291405]]

=== 2021-09-17 ===
* 11:35 arturo: [codfw1dev] install manila on cloudcontrol2001-dev ([[phab:T291257|T291257]])

=== 2021-09-16 ===
* 15:56 bstorm: removing downtime for labstore1005 so we'll know if it has another issue [[phab:T290318|T290318]]

=== 2021-09-09 ===
* 22:03 bstorm: restarted the prometheus-mysqld-exporter@s1 service as it was not working [[phab:T290630|T290630]]
* 03:15 bstorm: resetting swap on clouddb1017 [[phab:T290630|T290630]]
* 03:08 andrewbogott: stopping maintain-dbusers on labstore1004 for help diagnosing [[phab:T290630|T290630]]

=== 2021-09-03 ===
* 15:34 bstorm: rebooting labstore1005 to disconnect the drives from labstore1004 [[phab:T290318|T290318]]
* 15:24 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2002 [[phab:T290318|T290318]]
* 15:20 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2001 [[phab:T290318|T290318]]

=== 2021-08-30 ===
* 16:16 wm-bot: Added 1 new OSDs ['cloudcephosd1018.eqiad.wmnet'] - cookbook ran by andrew@buster
* 16:16 wm-bot: Added OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:10 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:07 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:07 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:07 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster

=== 2021-08-27 ===
* 18:57 andrewbogott: raising toolsbeta ram/core/instances quotas so majavah can experiment with bullseye

=== 2021-08-25 ===
* 14:45 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 14:42 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 14:42 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 14:42 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 14:41 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster

=== 2021-08-19 ===
* 17:39 bstorm: restarting glance image backup to try and clear the page

=== 2021-08-18 ===
* 16:21 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:21 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:21 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:17 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:16 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:15 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 14:47 andrewbogott: adding clouvirt1038 to the ceph aggregate, removing from the maintenance aggregate [[phab:T276922|T276922]]

=== 2021-08-17 ===
* 15:11 andrewbogott: rebooting cloudcephosd1008 to force raid rebuild -- [[phab:T287838|T287838]]

=== 2021-08-11 ===
* 13:51 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:48 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:47 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 13:47 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-08-10 ===
* 15:15 andrewbogott: restarting all designate services in eqiad1
* 15:04 andrewbogott: restarting designate-sink in eqiad1; it's complaining about rabbit but I don't want to restart rabbit yet

=== 2021-08-05 ===
* 09:37 dcaro: Taking one osd daemon down ot codfw cluster ([[phab:T288203|T288203]])

=== 2021-08-04 ===
* 19:20 bd808: Running deleteBatch.php on cloudweb2001-dev to remove legacy Heira: pages from labtestwiki

=== 2021-08-03 ===
* 17:40 bstorm: rerunning the glance backup script after failure

=== 2021-07-31 ===
* 00:10 andrewbogott: &quot;systemctl reset-failed cloud-init.service&quot; on all VMs for [[phab:T287309|T287309]]
* 00:08 andrewbogott: &quot;systemctl reset-failed cloud-final.service&quot; on all VMs for [[phab:T287309|T287309]]

=== 2021-07-27 ===
* 21:32 andrewbogott: putting cloudvirt1012 back into service [[phab:T286748|T286748]]
* 20:52 andrewbogott: draining VMs off of cloudvirt1012 so we can replace the battery for [[phab:T286748|T286748]]
* 15:15 andrewbogott: &quot;rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list&quot; cloud-wide

=== 2021-07-23 ===
* 15:22 bstorm: update wikireplicas-dns for s7 fix for web replicas

=== 2021-07-20 ===
* 17:07 andrewbogott: reloading haproxy on dbproxy1018 for [[phab:T286598|T286598]]
* 15:45 arturo: failback from labstore1006 to labstore1007 (dumps NFS) https://gerrit.wikimedia.org/r/c/operations/puppet/+/705417
* 00:10 bstorm: restarting nova-api on cloudcontrol1003 to try and recover whatever it's doing with designate_floating_ip_ptr_records_updater

=== 2021-07-19 ===
* 22:05 bstorm: set downtime scheduled for tomorrow from 1300 to 1600 UTC for cloudstore1008 and 1009 [[phab:T286599|T286599]]
* 20:40 andrewbogott: reloading haproxy on dbproxy1018 for [[phab:T286598|T286598]]
* 13:50 andrewbogott: upgrading mariadb to 10.3.29 on all cloudcontrols

=== 2021-07-16 ===
* 09:55 dcaro: checking HP raid issues on coludvirt1012 ([[phab:T286766|T286766]])

=== 2021-07-14 ===
* 21:08 andrewbogott: restarting lots of openstack services while trying to resolve [[phab:T286675|T286675]]
* 12:17 dcaro: doing ceph outage tests on codfw1 (fyi)

=== 2021-07-13 ===
* 10:57 dcaro: enabled autoscaling on codfw1 ceph cluster, setting a minimum of pgs on codfw1dev-compute to 128

=== 2021-07-02 ===
* 10:12 wm-bot: The cluster is not rebalance after adding the new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:12 wm-bot: Added 2 new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:12 wm-bot: Added OSD cloudcephosd1020.eqiad.wmnet... (2/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:10 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding OSD cloudcephosd1020.eqiad.wmnet... (2/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Added OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:02 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:02 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:01 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-07-01 ===
* 16:27 bstorm: failed over cloudstore1009 to cloudstore1008 [[phab:T224747|T224747]]
* 16:18 bstorm: downtimed cloudstore1008 and cloudstore1009 to fail over [[phab:T224747|T224747]]
* 14:25 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (2/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot: Added OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:20 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:20 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Added new OSD node cloudcephosd1016.eqiad.wmnet ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 11:13 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:58 dcaro: rebooting cloudcephosd1016 ([[phab:T285858|T285858]])
* 10:47 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:41 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-30 ===
* 21:48 bstorm: downtimed space alerts for scratch on cloudstore1008 until after the migration

=== 2021-06-25 ===
* 15:28 andrewbogott: restarting openstack services on cloudcontrol1005
* 09:16 arturo: icinga downtime cloudcontrols for 2h
* 08:20 dcaro: restarting rabbitmq on cloudcontrol100&lt;nowiki&gt;{&lt;/nowiki&gt;3,4&lt;nowiki&gt;}&lt;/nowiki&gt;

=== 2021-06-21 ===
* 13:54 dcaro: puppet fix merged and deployed, servers are back to normal
* 13:20 dcaro: merged broken puppet patch, downtimed all cloudvirts for 2h while fixing (nothing big, just added a bad systemd timer)

=== 2021-06-20 ===
* 22:21 andrewbogott: clearing admin-monitoring VMs; puppet has been failing lately due to a full drive on the puppetmaster

=== 2021-06-15 ===
* 01:18 bstorm: running a modified version of the prometheus dir size cron in screen [[phab:T284964|T284964]]

=== 2021-06-14 ===
* 10:13 dcaro: setting ssd to debug mode on tools-sgeexec-0917 ([[phab:T284130|T284130]])

=== 2021-06-10 ===
* 10:58 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:58 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:55 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:55 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:52 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:52 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:49 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:49 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:48 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:48 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:45 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:45 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:39 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:38 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:35 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:35 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:32 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:32 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:29 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:26 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:26 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:24 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:24 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-09 ===
* 17:33 arturo: removed icinga downtime for cloudmetrics1002 -- to see if hardware is healthy ([[phab:T281881|T281881]])
* 13:30 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:30 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-08 ===
* 23:19 bd808: Downtimed cloudmetrics1002 in icinga until 2021-06-30 23:59:01 ([[phab:T281881|T281881]])
* 21:08 bstorm: downtiming grafana-labs for maintenance
* 16:28 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:27 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:24 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:24 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:22 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:21 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:18 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:17 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

=== 2021-06-07 ===
* 14:27 andrewbogott: moving cloudvirt1040 from 'maintenance' aggregate to 'ceph' aggregate  [[phab:T281399|T281399]]

=== 2021-06-01 ===
* 13:12 dcaro: Changed the ceph osd_memory_target on eqiad pool to 6Gi (we were reaching the limit, swapping at some points)
* 09:57 arturo: fix PTR record for 185.15.56.1 ([[phab:T284025|T284025]])
* 09:56 arturo: fix PTR record for 185.15.56.1 ([[phab:T248025|T248025]])

=== 2021-05-27 ===
* 14:58 wm-bot: Testing - cookbook ran by dcaro@vulcanus

=== 2021-05-26 ===
* 19:10 andrewbogott: reimaging cloudvirt1018 to support local VM storage
* 18:07 andrewbogott: draining cloudvirt1018, converting it to a local-storage  host like cloudvirt1019 and 1020 -- [[phab:T283296|T283296]]
* 14:36 dcaro: Enabled syslog logging for osd.55 on eqiad ceph cluster for testing ([[phab:T281247|T281247]])
* 14:36 dcaro: Enabled syslog logging on codfw ceph cluster (mon/osd/mgr) ([[phab:T281247|T281247]])
* 11:26 arturo: [codfw1dev] purge old kernel packages in cloudvirt200[12]-dev
* 11:03 arturo: created public flavor `g3.cores16.ram36.disk20` (even though it was requested as private in [[phab:T283293|T283293]], but may be useful for others)

=== 2021-05-25 ===
* 16:14 bd808: Closed #wikimedia-cloud-admin on f***node
* 16:11 bd808: Closed #wikimedia-cloud-feed on f***node
* 15:19 dcaro: rebooted cloudvirt1020, starting VMs ([[phab:T275893|T275893]])
* 15:13 dcaro: rebooting cloudvirt1020 ([[phab:T275893|T275893]])
* 14:42 dcaro: taking cloudvirt1020 out for maintenance (openstack wise) so no new VMs are scheduled on it ([[phab:T275893|T275893]])

=== 2021-05-24 ===
* 22:32 andrewbogott: changing the default ttl for eqiad1.wikimedia.cloud. from 3600 to 60; this should help us avoid madness when re-using hostnames.
* 11:20 arturo: created `g3.cores2.ram80.disk40.private` for the wmf-research-tools project, to allow resizing a 40G disk instance

=== 2021-05-22 ===
* 02:14 bstorm: downtiming SMART alerts on dumps server labstore1007 for the weekend because it has been flapping [[phab:T281045|T281045]]

=== 2021-05-13 ===
* 21:25 bstorm: converted the maps and scratch volumes on cloudstore1008 (standby) to drbd [[phab:T224747|T224747]]
* 15:45 bstorm: re-running wikireplicas-dns after refactor of config to make sure it doesn't change anything

=== 2021-05-12 ===
* 14:23 arturo: [codfw1dev] cleanup old unused agents (bgp, ovs)
* 11:37 arturo: [codfw1dev] replacing cloudnet2003-dev with cloudnet2004-dev ([[phab:T281381|T281381]])

=== 2021-05-11 ===
* 18:00 andrewbogott: adding 'trove' service project in advance of deploying trove in eqiad1
* 10:22 arturo: rebooted cloudgw1002 (active) thus causing a failover to cloudgw1001

=== 2021-05-09 ===
* 10:53 arturo: icinga-downtime cloudmetrics1002 for 3 months ([[phab:T275605|T275605]])

=== 2021-05-07 ===
* 13:51 andrewbogott: add inherited 'admin' right to novaadmin user throughout eqiad1.  I was trying to narrow down the rights here but lack of admin breaks some workflows, e.g. [[phab:T281894|T281894]] and [[phab:T282235|T282235]]

=== 2021-05-06 ===
* 15:31 arturo: about to migrating CloudVPS network to the cloudgw architecture [[phab:T270704|T270704]]
* 11:14 dcaro: restarting cinder-volume on the eqiad control nodes to refresh the ceph libraries ([[phab:T282109|T282109]])

=== 2021-05-05 ===
* 16:07 dcaro: disallowing insecure global ids on the eqiad ceph cluster ([[phab:T280641|T280641]])
* 15:15 wm-bot: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:11 wm-bot: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:11 wm-bot: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:40 wm-bot: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:39 wm-bot: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:10 wm-bot: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 12:35 wm-bot: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:47 wm-bot: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:47 wm-bot: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:21 wm-bot: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:21 wm-bot: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:45 wm-bot: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:45 wm-bot: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:02 wm-bot: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus

=== 2021-05-04 ===
* 16:05 wm-bot: Safe reboot of 'cloudvirt1028.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:45 wm-bot: Safe rebooting 'cloudvirt1028.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:44 wm-bot: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:22 wm-bot: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:19 wm-bot: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:15 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 13:19 dcaro: rebooting cloudmetrics1002, got stuck again ([[phab:T275605|T275605]])
* 10:04 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:10 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:10 wm-bot: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:34 wm-bot: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:20 wm-bot: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:03 wm-bot: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus

=== 2021-05-03 ===
* 23:53 bstorm: running `maintain-dbusers harvest-replicas` on labstore1004 [[phab:T281287|T281287]]
* 23:51 bstorm: running `maintain-dbusers harvest-replicas` on labstore1004
* 16:34 wm-bot: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 16:29 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:41 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:41 wm-bot: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:31 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T280641|T280641]] - cookbook ran by dcaro@vulcanus)
* 10:23 wm-bot: (from a cookbook)
* 09:12 dcaro: draining and rebooting coludvirt1021 ([[phab:T280641|T280641]])
* 08:26 dcaro: draining and rebooting coludvirt1018 ([[phab:T280641|T280641]])

=== 2021-04-30 ===
* 11:16 dcaro: draining and rebooting coludvirt1017, last one today ([[phab:T280641|T280641]])
* 10:37 dcaro: draining coludvirt1016 for reboot ([[phab:T280641|T280641]])
* 09:48 dcaro: draining coludvirt1013 for reboot ([[phab:T280641|T280641]])

=== 2021-04-29 ===
* 15:11 dcaro: hard rebooting cloudmetrics1002, got hung again ([[phab:T275605|T275605]])
* 07:53 dcaro: Upgrading ceph libraries on cloudcontrol1005 to octopus ([[phab:T274566|T274566]])
* 07:51 dcaro: Upgrading ceph libraries on cloudcontrol1003 to octopus ([[phab:T274566|T274566]])
* 07:50 dcaro: Upgrading ceph libraries on cloudcontrol1004 to octopus ([[phab:T274566|T274566]])

=== 2021-04-28 ===
* 21:11 andrewbogott: cleaning up more references to deleted hypervisors with delete from services where topic='compute' and version != 53;
* 20:48 andrewbogott: cleaning up references to deleted hypervisors with mysql:root@localhost [nova_eqiad1]&gt; delete from compute_nodes where hypervisor_version != '5002000';
* 19:40 andrewbogott: putting cloudvirt1040 into the maintenance aggregate pending more info about [[phab:T281399|T281399]]
* 18:11 andrewbogott: adding cloudvirt1040, 1041 and 1042 to the 'ceph' host aggregate -- [[phab:T275081|T275081]]
* 11:06 dcaro: All ceph server side upgraded to Octopus! \o/ ([[phab:T280641|T280641]])
* 10:57 dcaro: Got a PG getting stuck on 'remapping' after the OSD came up, had to unset the norebalance and then set it again to get it unstuck ([[phab:T280641|T280641]])
* 10:34 dcaro: Slow/blocked opns from cloudcephmon03, &quot;osd_failure(failed timeout osd.32...&quot; (cloudcephosd1005), unset the cluster noout/norebalance and went away in a few secs, setting it again and continuing... ([[phab:T280641|T280641]])
* 09:03 dcaro: Waiting for slow heartbeats from osd.58(cloudcephosd1002) to recover... ([[phab:T280641|T280641]])
* 08:59 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) all from osd.58, currently on cloudcephosd1002 ([[phab:T280641|T280641]])
* 08:58 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) all from osd.58 ([[phab:T280641|T280641]])
* 08:58 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) ([[phab:T280641|T280641]])
* 08:21 dcaro: Upgrading all the ceph osds on eqiad ([[phab:T280641|T280641]])
* 08:21 dcaro: The clock skew seems intermittent, there's another task to follw it [[phab:T275860|T275860]] ([[phab:T280641|T280641]])
* 08:18 dcaro: All equiad ceph mons and mgrs upgraded ([[phab:T280641|T280641]])
* 08:18 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, cloudcephmon1001, they are back ([[phab:T280641|T280641]])
* 08:15 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, it went away, I'm guessing systemd-timesyncd fixed it ([[phab:T280641|T280641]])
* 08:14 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, looking ([[phab:T280641|T280641]])
* 07:58 dcaro: Upgrading ceph services on eqiad, starting with mons/managers ([[phab:T280641|T280641]])

=== 2021-04-27 ===
* 14:10 dcaro: codfw.openstack upgraded ceph libraries to 15.2.11 ([[phab:T280641|T280641]])
* 13:07 dcaro: codfw.openstack cloudvirt2002-dev done, taking cloudvirt2003-dev out to upgrade ceph libraries ([[phab:T280641|T280641]])
* 13:00 dcaro: codfw.openstack cloudvirt2001-dev back online, taking cloudvirt2002-dev out to upgrade ceph libraries ([[phab:T280641|T280641]])
* 10:51 dcaro: ceph.eqiad: cinder pool got it's pg_num increased to 1024, re-shuffle started ([[phab:T273783|T273783]])
* 10:48 dcaro: ceph.eqiad: Tweaked the target_size_ratio of all the pools, enabling autoscaler (it will increase cinder pool only) ([[phab:T273783|T273783]])
* 09:14 dcaro: manually force stopping the server puppetmaster-01 to unblock migration (in codfw1)
* 09:14 dcaro: manually force stopping the server puppetmaster-01 to unblock migration
* 08:59 dcaro: manually force stopping the server exploding-head on codfw, to try cold migration
* 08:47 dcaro: restarting nova-compute on cloudvirt2001-dev after upgrading ceph libraries to 15.2.11

=== 2021-04-26 ===
* 20:56 andrewbogott: deleting spurious 'codfw1dev' and 'codw1dev-4' regions in the dallas deployment; regions without endpoints break a bunch of things
* 09:45 dcaro: draining cloudvirt2001-dev with the new cookbooks ([[phab:T280641|T280641]])

=== 2021-04-23 ===
* 13:49 dcaro: testing the drain_cloudvirt cookbook on codfw1 openstack cluster, draining cloudvirt2001 ([[phab:T280641|T280641]])
* 11:12 dcaro: testing the drain_cloudvirt cookbook on codfw1 openstack cluster ([[phab:T280641|T280641]])
* 09:32 dcaro: finished upgrade of ceph cluster on codfw1 using exclusively cookbooks ([[phab:T280641|T280641]])
* 09:17 dcaro: testing the upgrade_osds cookbook on codfw1 ceph cluster ([[phab:T280641|T280641]])
* 08:17 dcaro: testing the upgrade_mons cookbook on codfw1 ceph cluster ([[phab:T280641|T280641]])

=== 2021-04-21 ===
* 17:59 dcaro: all monitors upgraded on codfw1 with one cookbook `cookbook --verbose -c ~/.config/spicerack/cookbook.yaml wmcs.ceph.upgrade_mons --monitor-node-fqdn cloudcephmon2002-dev.codfw.wmnet` ([[phab:T280641|T280641]])
* 17:47 dcaro: upgrading monitors and mrg nodes on codfw ceph cluster ([[phab:T280641|T280641]])
* 13:26 dcaro: testing ceph upgrade cookbook on cloudcephmon2002-dev ([[phab:T280641|T280641]])

=== 2021-04-20 ===
* 20:21 andrewbogott: reboot cloudservices1003
* 20:13 andrewbogott: reboot cloudservices1004

=== 2021-04-19 ===
* 08:40 dcaro: enabling puppet on labstore1004 after mysql restart ([[phab:T279657|T279657]])
* 08:09 dcaro: downtiming labstore1004 and stopping puppet for mysql restart ([[phab:T279657|T279657]])

=== 2021-04-14 ===
* 10:48 dcaro: Upgrade of codfw ceph to octopus 15.2.20 done, will run some performance tests now ([[phab:T274566|T274566]])
* 10:41 dcaro: Upgrade of codfw ceph to octopus 15.2.20, mgrs upgraded, osds next ([[phab:T274566|T274566]])
* 10:37 dcaro: Upgrade of codfw ceph to octopus 15.2.20, mons upgraded, mgrs next ([[phab:T274566|T274566]])
* 10:15 dcaro: starting the upgrade of codfw ceph to octopus 15.2.20 ([[phab:T274566|T274566]])
* 10:07 dcaro: Merged the ceph 15 (Octopus) repo deployment to codfw, only the repo, not the packages ([[phab:T274566|T274566]])

=== 2021-04-13 ===
* 16:42 dcaro: Ceph balancer got the cluster to eval 0.014916, that is 88-77% usage for compute pool, and 28-19% usage for the cinder one \o/ ([[phab:T274573|T274573]])
* 15:08 dcaro: Activating continuous upmap balancer, keeping a close eye ([[phab:T274573|T274573]])
* 15:03 dcaro: Executing a second pass, there's still movements to improve the eval of 0.030075 ([[phab:T274573|T274573]])
* 15:02 dcaro: First pass finished, improved eval to 0.030075 ([[phab:T274573|T274573]])
* 14:49 dcaro: Running the first_pass balancing plan on ceph eqiad, current eval 0.030622 ([[phab:T274573|T274573]])
* 14:43 dcaro: enabling ceph upmap pg balancer on equiad ([[phab:T274573|T274573]])
* 14:36 andrewbogott: upgrading codfw1dev to version Victoria, [[phab:T261137|T261137]]
* 13:11 andrewbogott: upgrading eqiad1 designate to version Victoria, [[phab:T261137|T261137]]
* 10:44 dcaro: enabled ceph upmap balancer on codfw ([[phab:T274573|T274573]],[[phab:T274573|T274573]])

=== 2021-04-07 ===
* 21:33 andrewbogott: upgrading codfw1dev designate to Victoria

=== 2021-04-04 ===
* 17:36 andrewbogott: upgrading eqiad1 designate to Ussuri

=== 2021-04-02 ===
* 14:12 andrewbogott: upgrading codfw1dev to OpenStack version Ussuri

=== 2021-04-01 ===
* 12:15 dcaro: Restoring the 4.9 kernel on cloudcephosd2003-dev and upgrading ([[phab:T274565|T274565]])
* 10:29 dcaro: Done restoring the 4.9 kernel on cloudcephosd2001-dev and upgrading, requires logging into console to boot from the older kernel before removing the newer one ([[phab:T274565|T274565]])
* 10:10 dcaro: Restoring the 4.9 kernel on cloudcephosd2001-dev and upgrading ([[phab:T274565|T274565]])

=== 2021-03-31 ===
* 08:47 dcaro: upgrading cinder on codfw cloudcontrol2* nodes ([[phab:T278845|T278845]])

=== 2021-03-30 ===
* 09:53 arturo: rebooting cloudnet1003 to cleanup conntrack table, it wouldn't cleanup by hand ...

=== 2021-03-28 ===
* 15:42 andrewbogott: updated debian-10.0-buster base image

=== 2021-03-27 ===
* 09:54 arturo: cleanup conntrack table in qrouter nents in cloudnet1003 (backup)

=== 2021-03-25 ===
* 19:03 andrewbogott: deleting all unused (per wmcs-imageusage) Jessie base images from Glance
* 17:15 andrewbogott: refreshing puppet compiler facts for tools project
* 10:31 dcaro: kernel upgrade on osds on codfw done, running performance tests ([[phab:T274565|T274565]])
* 10:24 dcaro: upgrading kernel on cloudcephosd2003-dev and reboot ([[phab:T274565|T274565]])
* 10:18 dcaro: upgrading kernel on cloudcephosd2002-dev and reboot ([[phab:T274565|T274565]])
* 10:08 dcaro: upgrading kernel on cloudcephmon2003-dev and reboot ([[phab:T274565|T274565]])

=== 2021-03-24 ===
* 09:19 dcaro: restarted wmcs-backup on cloudvirt1024 as it failed due to an image being removed while running ([[phab:T276892|T276892]])

=== 2021-03-23 ===
* 11:33 arturo: root@cloudcontrol1005:~# wmcs-novastats-dnsleaks --delete

=== 2021-03-22 ===
* 10:10 arturo: cleanup conntrack table in standby node: aborrero@cloudnet1003:~ $ sudo ip netns exec qrouter-d93771ba-2711-4f88-804a-{{Gerrit|8df6fd03978a}} conntrack -F

=== 2021-03-19 ===
* 17:18 bstorm: running `ALTER TABLE account MODIFY COLUMN type ENUM('user','tool','paws');` against the labsdbaccounts database on m5 [[phab:T276284|T276284]]
* 14:29 andrewbogott: switching admin-monitoring project to use an upstream debian image; I want to see how this affects performance
* 00:30 bstorm: downtimed labstore1004 to check some things in debug mode

=== 2021-03-17 ===
* 17:28 bstorm: restarted the backup-glance-images job to clear errors in systemd [[phab:T271782|T271782]]
* 17:16 andrewbogott: set default cinder quota for projects to 80Gb with &quot;update quota_classes set hard_limit=80 where resource='gigabytes';&quot; on database 'cinder'
* 16:58 andrewbogott: disabling all flavors with &gt;20Gb root storage with &quot;update flavors set disabled=1 where root_gb&gt;20;&quot; in nova_eqiad1_api

=== 2021-03-10 ===
* 16:51 arturo: rebooting cloudvirt1030 for [[phab:T275753|T275753]]
* 13:14 dcaro: starting manually the canary VM for cloudvirt1029 (nova start 349830f6-3b39-4a8c-ada4-{{Gerrit|a7439f65cffe}}) ([[phab:T275753|T275753]])
* 12:51 arturo: draining cloudvirt1030 for [[phab:T275753|T275753]]
* 12:47 arturo: rebooting cloudvirt1029 for [[phab:T275753|T275753]]
* 11:56 arturo: [codfw1dev] restart rabbitmq-server in all 3 cloudcontrol servers for [[phab:T276964|T276964]]
* 11:53 arturo: [codfw1dev] restart nova-conductor in all 3 cloudcontrol servers for [[phab:T276964|T276964]]
* 11:31 arturo: draining cloudvirt1029 for [[phab:T275753|T275753]]
* 11:29 arturo: rebooting cloudvirt1013 for [[phab:T275753|T275753]]
* 11:05 arturo: draining cloudvirt1013 for [[phab:T275753|T275753]]
* 11:00 arturo: rebooting cloudvirt1028 for [[phab:T275753|T275753]]
* 10:33 arturo: draining cloudvirt1028 for [[phab:T275753|T275753]]
* 10:29 arturo: rebooting cloudvirt1023 for [[phab:T275753|T275753]]
* 09:37 arturo: draining cloudvirt1023 for [[phab:T275753|T275753]]
* 09:07 arturo: [codfw1dev] reimaging cloudvirt2003-dev ([[phab:T276964|T276964]])

=== 2021-03-09 ===
* 16:27 arturo: rebooting cloudvirt1027 ([[phab:T275753|T275753]])
* 13:39 arturo: draining cloudvrit1027 for [[phab:T275753|T275753]]
* 13:35 arturo: icinga-downtime cloudvirt1038 for 30 days for [[phab:T276922|T276922]]
* 13:21 arturo: add cloudvirt1039 to the ceph host aggregate (no longer a spare, we have cloudvirt1038 with HW failures)
* 12:52 arturo: cloudvirt1038 hard powerdown / powerup for [[phab:T276922|T276922]]
* 12:33 arturo: rebooting cloudvirt1038 ([[phab:T275753|T275753]])
* 10:58 arturo: draining cloudvirt1038 ([[phab:T275753|T275753]])
* 10:54 arturo: rebooting cloudvirt1037 ([[phab:T275753|T275753]])
* 09:59 arturo: draining cloudvirt1037 ([[phab:T275753|T275753]])
* 09:12 dcaro: restarted the wmcs-backup service on cloudvirt1024 to retry the backups (failed because a VM was removed in-between, [[phab:T276892|T276892]])

=== 2021-03-05 ===
* 21:40 andrewbogott: replacing 'observer' role with 'reader' role in eqiad1 [[phab:T276018|T276018]]
* 21:21 andrewbogott: replacing 'observer' role with 'reader' role in eqiad1
* 16:23 arturo: rebooting cloudvirt1036 for [[phab:T275753|T275753]]
* 12:30 arturo: draining cloudvirt1036 for [[phab:T275753|T275753]]
* 12:25 arturo: rebooting cloudvirt1035 for [[phab:T275753|T275753]]
* 10:49 arturo: rebooting cloudvirt1035 for [[phab:T275753|T275753]]
* 10:47 arturo: rebooting cloudvirt1034 for [[phab:T275753|T275753]]
* 10:26 arturo: draining cloudvirt1034 for [[phab:T275753|T275753]]
* 10:25 arturo: rebooting cloudvirt1033 for [[phab:T275753|T275753]]
* 09:18 arturo: draining cloudvirt1033 for [[phab:T275753|T275753]]

=== 2021-03-04 ===
* 18:36 andrewbogott: rebooting cloudmetrics1002; the console is hanging
* 16:59 arturo: rebooting cloudvirt1032 for [[phab:T275753|T275753]]
* 16:34 arturo: draining cloudvirt1032 for [[phab:T275753|T275753]]
* 16:33 arturo: rebooting cloudvirt1031 for [[phab:T275753|T275753]]
* 16:11 arturo: draining cloudvirt1031 for [[phab:T275753|T275753]]
* 16:09 arturo: rebooting cloudvirt1026 for [[phab:T275753|T275753]]
* 15:57 arturo: draining cloudvirt1026 for [[phab:T275753|T275753]]
* 15:55 arturo: rebooting cloudvirt1025 for [[phab:T275753|T275753]]
* 15:41 arturo: draining cloudvirt1025 for [[phab:T275753|T275753]]
* 15:12 arturo: rebooting cloudvirt1024 for [[phab:T275753|T275753]]
* 11:29 arturo: draining cloudvirt1024 for [[phab:T275753|T275753]]
* 11:24 dcaro: rebooted cloudvirt1022, re-adding to ceph and removing from maintenance host aggregate for [[phab:T275753|T275753]]
* 11:01 dcaro: rebooting cloudvirt1022 for [[phab:T275753|T275753]]
* 09:12 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]

=== 2021-03-03 ===
* 17:16 andrewbogott: restarting rabbitmq-server on cloudcontrol1003,1004,1005; trying to explain amqp errors in scheduler logs
* 16:03 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]
* 16:03 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]
* 16:00 arturo: move cloudvirt1013 into the 'toobusy' host aggregate, it has 221% cpu subscription and 82% MEM subscription
* 15:34 arturo: rebooting cloudvirt1021 for [[phab:T275753|T275753]]
* 14:31 arturo: draining cloudvirt1021 for [[phab:T275753|T275753]]
* 13:59 arturo: rebooting cloudvirt1018 for [[phab:T275753|T275753]]
* 13:28 arturo: draining cloudvirt1018 for [[phab:T275753|T275753]]
* 12:49 arturo: rebooting cloudvirt1017 for [[phab:T275753|T275753]]
* 12:22 arturo: draining cloudvirt1017 for [[phab:T275753|T275753]]
* 12:20 arturo: rebooting cloudvirt1016 for [[phab:T275753|T275753]]
* 12:01 arturo: draining cloudvirt1016 for [[phab:T275753|T275753]]
* 11:59 arturo: cloudvirt1014 now in the ceph host aggregate
* 11:58 arturo: rebooting cloudvirt1014 for [[phab:T275753|T275753]]
* 11:50 arturo: moved cloudvirt1023 away from the maintenance host aggregate, leave it in the ceph aggregate (was in the 2)
* 11:47 arturo: moved cloudvirt1014 to the 'maintenance' host aggregate, drain it for [[phab:T275753|T275753]]
* 10:01 arturo: icinga-downtime cloudnet1003 for 14 days bc potential alerting storm due to firmware issues ([[phab:T271058|T271058]])
* 10:01 arturo: rebooting again cloudnet1003 (no network failover) ([[phab:T271058|T271058]])
* 09:59 arturo: update firmware-bnx2x from 20190114-2 to 20200918-1~bpo10+1 on cloudnet1003 ([[phab:T271058|T271058]])
* 09:30 arturo: installing linux kernel 5.10.13-1~bpo10+1 in cloudnet1003 and rebooting it (network failover) ([[phab:T271058|T271058]])

=== 2021-03-02 ===
* 17:16 andrewbogott: rebooting cloudvirt1039 to see if I can trigger [[phab:T276208|T276208]]
* 16:10 arturo: [codfw1dev] restart nova-compute on cloudvirt2002-dev
* 11:59 arturo: moved cloudvirt1012 to 'maintenance' host aggregate. Drain it with `wmcs-drain-hypervisor` to reboot it for [[phab:T275753|T275753]]
* 11:59 arturo: cloudvirt1023 is affected by [[phab:T276208|T276208]] and cannot be rebooted. Put it back into the ceph hos aggregate
* 10:43 arturo: moved cloudvirt1013 cloudvirt1032 cloudvirt1037 back into the 'ceph' host aggregate
* 10:13 arturo: moved cloudvirt1023 to 'maintenance' host aggregate. Drain it with `wmcs-drain-hypervisor` to reboot it for [[phab:T275753|T275753]]

=== 2021-03-01 ===
* 20:12 andrewbogott: removing novaadmin from all projects save 'admin' for [[phab:T274385|T274385]]
* 19:51 andrewbogott: removing novaobserver from all projects save 'observer' for [[phab:T274385|T274385]]
* 19:50 andrewbogott: adding inherited domain-wide roles to novaadmin and novaobserver as per [[phab:T274385|T274385]]

=== 2021-02-28 ===
* 04:54 andrewbogott: restarted redis-server on tools-redis-1003 and tools-redis-1004 in an attempt to reduce replag, no real change detected

=== 2021-02-27 ===
* 00:33 andrewbogott: sudo cumin --timeout 500 &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i buster &amp;&amp; uname -r  {{!}} grep -v 4.19.0-14-amd64 &amp;&amp; reboot'
* 00:28 andrewbogott: sudo cumin --timeout 500 &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i buster &amp;&amp; uname -r  {{!}} grep -v 4.19.0-14-amd64 &amp;&amp; echo reboot'
* 00:09 andrewbogott: sudo cumin &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i stretch &amp;&amp; uname -r  {{!}} grep -v 4.19.0-0.bpo.14-amd64 &amp;&amp; reboot'

=== 2021-02-26 ===
* 14:58 dcaro: [eqiad] rebooting cloudcephosd1015 (last osd \o/) for kernel upgrade ([[phab:T275753|T275753]])
* 14:51 dcaro: [eqiad] rebooting cloudcephosd1014 for kernel upgrade ([[phab:T275753|T275753]])
* 14:44 dcaro: [eqiad] rebooting cloudcephosd1013 for kernel upgrade ([[phab:T275753|T275753]])
* 14:38 dcaro: [eqiad] rebooting cloudcephosd1012 for kernel upgrade ([[phab:T275753|T275753]])
* 14:31 dcaro: [eqiad] rebooting cloudcephosd1011 for kernel upgrade ([[phab:T275753|T275753]])
* 14:25 dcaro: [eqiad] rebooting cloudcephosd1010 for kernel upgrade ([[phab:T275753|T275753]])
* 14:17 dcaro: [eqiad] rebooting cloudcephosd1009 for kernel upgrade ([[phab:T275753|T275753]])
* 13:54 dcaro: [eqiad] downtimed alert1001 Ceph OSDs down alert until 18:00 GMT+1 as that is not under the host being rebooted ([[phab:T275753|T275753]])
* 13:51 dcaro: [eqiad] rebooting cloudcephosd1008 for kernel upgrade ([[phab:T275753|T275753]])
* 13:45 dcaro: [eqiad] rebooting cloudcephosd1007 for kernel upgrade ([[phab:T275753|T275753]])
* 13:38 dcaro: [eqiad] rebooting cloudcephosd1006 for kernel upgrade ([[phab:T275753|T275753]])
* 12:07 dcaro: [eqiad] rebooting cloudcephosd1005 for kernel upgrade ([[phab:T275753|T275753]])
* 12:00 arturo: rebooting cloudcontrol1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:42 arturo: rebooting cloudcontrol1004 for kernel upgrade ([[phab:T275753|T275753]])
* 11:41 dcaro: [eqiad] rebooting cloudcephosd1004 for kernel upgrade ([[phab:T275753|T275753]])
* 11:32 dcaro: [eqiad] rebooting cloudcephosd1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:30 arturo: rebooting cloudcontrol1005 for kernel upgrade ([[phab:T2|T2]]
* 11:26 dcaro: [eqiad] rebooting cloudcephosd1002 for kernel upgrade ([[phab:T275753|T275753]])
* 11:16 dcaro: [eqiad] rebooting cloudcephosd1001 for kernel upgrade ([[phab:T275753|T275753]])
* 11:11 dcaro: [eqiad] rebooting cloudcephmon1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:05 dcaro: [eqiad] rebooting cloudcephmon1002 for kernel upgrade ([[phab:T275753|T275753]])
* 10:59 dcaro: [eqiad] rebooting cloudcephmon1001 for kernel upgrade ([[phab:T275753|T275753]])
* 10:45 arturo: rebooting cloudvirt1039 into a new kernel ([[phab:T275753|T275753]]) --- spare
* 10:43 dcaro: [codfw1dev] rebooting cloudcephmon2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:38 dcaro: [codfw1dev] rebooting cloudcephmon2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:29 dcaro: [codfw1dev] rebooting cloudcephmon2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:24 arturo: [codfw1dev] purge old kernel packages on cloudvirt2003-dev to force boot into a new kernel ([[phab:T275753|T275753]])
* 10:11 arturo: [codfw1dev] manually creating /boot/grub/ on cloudvirt2003-dev to allow update-grub2 to run (so it can reboot into a new kernel) ([[phab:T275753|T275753]])
* 10:11 dcaro: [codfw1dev] rebooting cloudcephosd2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:05 dcaro: [codfw1dev] rebooting cloudcephosd2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:01 arturo: [codfw1dev] rebooting cloudvirt200X-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:59 arturo: [codfw1dev] rebooting cloudweb2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:53 arturo: [codfw1dev] rebooting cloudservices2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:51 arturo: [codfw1dev] rebooting cloudservices2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:45 arturo: [codfw1dev] rebooting cloudcontrol2004-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:44 arturo: [codfw1dev] rebooting cloudbackup[2001-2002].codfw.wmnet for kernel upgrade ([[phab:T275753|T275753]])
* 09:43 dcaro: [codfw1dev] rebooting cloudcephosd2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:41 arturo: [codfw1dev] rebooting cloudcontrol2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:33 arturo: [codfw1dev] rebooting cloudcontrol2001-dev for kernel upgrade ([[phab:T275753|T275753]])

=== 2021-02-25 ===
* 14:56 arturo: deployed wmcs-netns-events daemon to all cloudnet servers ([[phab:T275483|T275483]])

=== 2021-02-24 ===
* 11:07 arturo: force-reboot cloudmetrics1002, add icinga downtime for 2 hours. Investigating some server issue
* 00:17 bstorm: set  --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main stretch image in glance on eqiad1 [[phab:T275430|T275430]]

=== 2021-02-23 ===
* 22:43 bstorm: set  --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main buster image in glance on eqiad1 [[phab:T275430|T275430]]
* 20:36 andrewbogott: adding r/o access to the eqiad1-glance-images ceph pool for the client.eqiad1-compute for [[phab:T275430|T275430]]
* 10:49 arturo: rebooting clounet1004 into new kernel from buster-bpo ([[phab:T271058|T271058]])
* 10:49 arturo: installing linux-image-amd64 from buster-bpo 5.10.13-1~bpo10+1 in cloudnet1004 ([[phab:T271058|T271058]])

=== 2021-02-22 ===
* 17:15 bstorm: restarting nova-compute on cloudvirt1016 and cloudvirt1036 in case it helps [[phab:T275411|T275411]]
* 15:02 dcaro: Re-uploaded the debian buster 10.0 image from rbd to glance, that worked, re-spawning all the broken instances ([[phab:T275378|T275378]])
* 11:12 dcaro: Refreshing all the canary instances ([[phab:T275354|T275354]])

=== 2021-02-18 ===
* 14:50 arturo: rebooting cloudnet1004 for [[phab:T271058|T271058]]
* 10:25 dcaro: Rebooting cloudmetrics1001 to apply new kernel ([[phab:T275116|T275116]])
* 10:16 dcaro: Rebooting cloudmetrics1002 to apply new kernel ([[phab:T275116|T275116]])
* 10:14 dcaro: Upgrading grafana on cloudmetrics1002 ([[phab:T275116|T275116]])
* 10:12 dcaro: Upgrading grafana on cloudmetrics1001 ([[phab:T275116|T275116]])

=== 2021-02-17 ===
* 15:58 arturo: deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/664845 to cloudnet servers ([[phab:T268335|T268335]])

=== 2021-02-15 ===
* 16:25 arturo: [codfw1dev] rebooting all cloudgw200x-dev / cloudnet200x-dev servers ([[phab:T272963|T272963]])
* 15:45 arturo: [codfw1dev] drop subnet definition for cloud-instances-transport1-b-codfw ([[phab:T272963|T272963]])
* 15:45 arturo: [codfw1dev] connect virtual router cloudinstances2b-gw to vlan cloud-gw-transport-codfw (185.15.57.10) ([[phab:T272963|T272963]])

=== 2021-02-11 ===
* 12:01 arturo: [codfw1dev] drop instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` (was buster, cannot use it yet)
* 11:59 arturo: [codfw1dev] create instance `tools-codfw1dev-bastion-2` (stretch) in `tools-codfw1dev` to test stuff related to [[phab:T272397|T272397]]
* 11:45 arturo: [codfw1dev] create instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` to test stuff related to [[phab:T272397|T272397]]
* 11:42 arturo: [codfw1dev] drop `tools` project, create `tools-codfw1dev`
* 11:38 arturo: [codfw1dev] drop `coudinfra` project (we are using `cloudinfra-codfw1dev` there)
* 05:37 bstorm: downtimed cloudnet1004 for another week [[phab:T271058|T271058]]

=== 2021-02-09 ===
* 15:23 arturo: icinga-downtime for 2h everything *labs *cloud for openstack upgrades
* 11:14 dcaro: Merged the osd scheduler change for all osds, applying on all cloudcephosd* ([[phab:T273791|T273791]])

=== 2021-02-08 ===
* 18:50 bstorm: enabled puppet on cloudvirt1023 for now [[phab:T274144|T274144]]
* 18:44 bstorm: restarted the backup_vms.service on cloudvirt1027 [[phab:T274144|T274144]]
* 17:51 bstorm: deleted project pki [[phab:T273175|T273175]]

=== 2021-02-05 ===
* 10:59 arturo: icinga-downtime labstore1004 tools share space check for 1 week ([[phab:T272247|T272247]])
* 10:21 dcaro: This was affecting maps and several others, maps and project-proxy have been fixed ([[phab:T273956|T273956]])
* 09:19 dcaro: Some certs around the infra are expired ([[phab:T273956|T273956]])

=== 2021-02-04 ===
* 10:12 dcaro: Increasing the memory limit of osds in eqiad from 8589934592(8G) to 12884901888(12G) ([[phab:T273851|T273851]])

=== 2021-02-03 ===
* 09:59 dcaro: Doing a full vm backup on cloudvirt1024 with the new script ([[phab:T260692|T260692]])
* 01:50 bstorm: icinga-downtime cloudnet1004 for a week [[phab:T271058|T271058]]

=== 2021-02-02 ===
* 17:14 dcaro: Changed osd memory limit from 4G to 8G ([[phab:T273649|T273649]])
* 11:00 arturo: icinga-downtime cloudvirt-wdqs1001 for 1 week ([[phab:T273579|T273579]])
* 03:12 andrewbogott: running /usr/local/sbin/wmcs-purge-backups and /usr/local/sbin/wmcs-backup-instances on cloudvirt1024 to see why the backup job paged

=== 2021-01-29 ===
* 15:36 andrewbogott: disabling puppet and some services on eqiad1 cloudcontrol nodes; replacing nova-placement-api with placement-api

=== 2021-01-28 ===
* 19:44 andrewbogott: shutting down cloudcontrol2001-dev because it's in a partially upgraded state; will revive when it's time for Train

=== 2021-01-27 ===
* 00:50 bstorm: icinga-downtime cloudnet1004 for a week [[phab:T271058|T271058]]

=== 2021-01-22 ===
* 16:44 andrewbogott: upgrading designate on cloudvirt1003/1004 to OpenStack 'train'
* 11:29 dcaro: Doing some tests removed cloudcontrol1003 puppet cert, regenerating...

=== 2021-01-21 ===
* 11:35 arturo: merging core router firewall changes https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657439 ([[phab:T209082|T209082]])
* 11:30 arturo: merging core router firewall changes https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657358 ([[phab:T272486|T272486]], [[phab:T209082|T209082]])

=== 2021-01-20 ===
* 10:49 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657302 ([[phab:T209082|T209082]])
* 10:05 dcaro: Everything looks ok, created a new vm with a volume in ceph without issues, and on warnings/errors on ceph status, closing ([[phab:T272303|T272303]])
* 09:55 dcaro: Eqiad ceph cluster uprgaded, doing sanity checks ([[phab:T272303|T272303]])
* 09:46 dcaro: 75% of the eqiad cluster upgraded... continuing ([[phab:T272303|T272303]])
* 09:37 dcaro: 25% of the eqiad cluster upgraded... continuing ([[phab:T272303|T272303]])
* 09:24 dcaro: Mgr daemons upgraded and running, upgrading osd daemons on servers cloudcephosd1*, this make take a bit longer ([[phab:T272303|T272303]])
* 09:22 dcaro: Mon daemons upgraded and running, upgrading mgr daemons on servers cloudcephmon1* ([[phab:T272303|T272303]])
* 09:16 dcaro: Starting eqiad ceph upgrade, upgrading the mon servers cloudcephmon1* ([[phab:T272303|T272303]])
* 09:01 dcaro: Will start the ceph upgrade in 15 min, no downtime nor performance impact is expected ([[phab:T272303|T272303]])

=== 2021-01-19 ===
* 10:17 arturo: icinga-downtime cloudnet1004 for 1 week ([[phab:T271058|T271058]])

=== 2021-01-18 ===
* 16:00 dcaro: Codfw1 ceph cluster uprgaded, will wait until tomorrow to see if there's any instability, but everything looks fine ([[phab:T272303|T272303]])
* 15:38 dcaro: Upgraded mgr sevices on codfw ceph cluster, starting with osd ones ([[phab:T272303|T272303]])
* 15:35 dcaro: Upgraded mon sevices on codfw ceph cluster, starting with mgr ones ([[phab:T272303|T272303]])
* 15:21 dcaro: Starting upgrade of ceph mon nodes on codfw ([[phab:T272303|T272303]])
* 15:06 dcaro: re-enabling puppet on cloudcephosd2* hosts
* 13:53 dcaro: disabling puppet on cloudcephosd2* to resume perf tests
* 10:50 dcaro: re-enabling puppet on cephcloudosd2* (codfw)
* 10:07 dcaro: disabling puppet on cephcloudosd2* (codfw) to do some performance tests
* 09:00 dcaro: Enabling custom application 'cinder' on pool codfw1dev-cinder to get rid of health warnings

=== 2021-01-17 ===
* 16:53 arturo: icinga downtime labstore1004 /srv/tools space check for 3 days ([[phab:T272247|T272247]])

=== 2021-01-15 ===
* 13:41 arturo: icinga downtime labstore1004 maintain-dbuser alert until 2021-01-19 ([[phab:T272125|T272125]])
* 09:47 arturo: labstore1004 maintain-dbusers affected by [[phab:T272127|T272127]] and [[phab:T272125|T272125]]
* 09:22 arturo: restart maintain-dbusers.service in labstore1004
* 08:19 dcaro: Merging the patch to disable write caches on ceph osds ([[phab:T271527|T271527]])

=== 2021-01-13 ===
* 17:03 arturo: remove cloudvirt1013 cloudvirt1032 cloudvirt1037 to the 'toobusy' host aggregate to prevent further CPU oversubscribing
* 12:40 arturo: try increasing systemd watchdog timeout for conntrackd in cloudnet1004 ([[phab:T268335|T268335]])
* 11:45 dcaro: https://gerrit.wikimedia.org/r/c/operations/puppet/+/654419 merged and deployed (and tested) ([[phab:T268877|T268877]])
* 11:40 dcaro: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/654419 that might affect the encapi service (puppet on cloud environment), no downtime expected though ([[phab:T268877|T268877]])
* 10:56 arturo: trying to cleanup dpkg package mess in cloudnet2002-dev
* 10:02 arturo: prevent floating IP allocation from neutron transport subnet: root@cloudcontrol1005:~# neutron subnet-update --allocation-pool start=185.15.56.244,end=185.15.56.244 cloud-instances-transport1-b-eqiad1 ([[phab:T271867|T271867]])

=== 2021-01-12 ===
* 10:33 arturo: reboot cloudnet1004
* 10:32 arturo: update firmware-bnx2x from 20190114-2 to 20200918-1~bpo10+1 on cloudnet1004 ([[phab:T271058|T271058]])

=== 2021-01-11 ===
* 10:22 arturo: doubling size of conntrack table in cloudnet servers https://gerrit.wikimedia.org/r/c/operations/puppet/+/655407 ([[phab:T271058|T271058]])
* 10:07 arturo: manually cleanup conntrack table in cloudnet1004 ([[phab:T271058|T271058]])
* 09:19 dcaro: cleaned up ~1800 snapshots, 109 remaining only, one for each host x image combination (plus some ephemeral ones while doing backups), closing the task ([[phab:T270478|T270478]])
* 08:39 dcaro: cleaning up dangling snapshots now that we have the new suffixed ones ([[phab:T270478|T270478]])

=== 2021-01-10 ===
* 16:02 andrewbogott: restarting rabbitmq-server on all eqiad1 cloudcontrols
* 15:54 andrewbogott: restating neutron-metadata-agent on cloudnet1004 due to many syslog complaints

=== 2021-01-08 ===
* 11:25 arturo: rebooting both cloudnet2002-dev/cloudnet2003-dev to make sure interfaces are set up correctl ([[phab:T271517|T271517]])
* 11:22 arturo: connecting cloudnet2002-dev cloudnet2003-dev back to vlan 2120 ([[phab:T271517|T271517]])
* 11:06 arturo: root@cloudcontrol2001-dev:~# openstack router set --external-gateway wan-transport-codfw --fixed-ip subnet=cloud-instances-transport1-b-codfw,ip-address=208.80.153.190 cloudinstances2b-gw ([[phab:T271517|T271517]])
* 11:02 arturo: root@cloudcontrol2001-dev:~# openstack router set --enable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw ([[phab:T271517|T271517]])
* 11:01 arturo: enabling neutron hacks in codfw1dev (cloudnet2002-dev, cloudnet2003-dev) ([[phab:T271517|T271517]])
* 10:55 arturo: aborrero@labtestvirt2003:~ $ sudo ifdown eno2.2107 ([[phab:T271517|T271517]])
* 10:55 arturo: aborrero@labtestvirt2003:~ $ sudo ifdown eno2.2120 ([[phab:T271517|T271517]])
* 10:53 arturo: root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 208.80.153.185 --ip-version 4 --network wan-transport-codfw --no-dhcp --subnet-range 208.80.153.184/29 cloud-instances-transport1-b-codfw ([[phab:T271517|T271517]])
* 10:40 dcaro: Finished tests, brining osd online (od.48) for eqiad ceph cluster ([[phab:T271417|T271417]])
* 09:59 dcaro: Started performance tests on sdc (od.48) for eqiad ceph cluster ([[phab:T271417|T271417]])
* 09:41 dcaro: Taking osd.48 from eqiad ceph cluster out to do performance tests ([[phab:T271417|T271417]])

=== 2021-01-07 ===
* 15:19 dcaro: Finished speed tests on cloudcephosd2001-dev, reprovisioning the osd.0 sdc ([[phab:T271417|T271417]])
* 14:39 dcaro: Starting speed tests on cloudcephosd2001-dev sdc ([[phab:T271417|T271417]])
* 12:54 dcaro: Taking osd.0 down on codfw ceph cluster to try the disk performance testing process ([[phab:T271417|T271417]])
* 11:35 arturo: merging dmz_cidr change ([[phab:T209082|T209082]], [[phab:T267779|T267779]])

=== 2021-01-05 ===
* 10:40 dcaro: removing dumps-[1..*] backups from cloudvirt1024 as they are not needed ([[phab:T271094|T271094]])

=== 2021-01-03 ===
* 07:06 dcaro: Got a network hiccup on cloudnet1004, keeping track here [[phab:T271058|T271058]]

=== 2020-12-28 ===
* 12:32 arturo: stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 ([[phab:T260692|T260692]])
* 12:32 arturo: stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 ([[phab:T260682|T260682]])
* 12:23 arturo: icinga downtime cloudvirt1026 disk space check until january 5 ([[phab:T260692|T260692]])
* 06:15 andrewbogott: restarting designate-central on cloudservices1003/1004.  I'm pretty sure they're distressed because of DB lag but it's worth a try

=== 2020-12-23 ===
* 15:38 andrewbogott: restarting rabbitmq on cloudcontrol1004; suspected leaks
* 15:33 andrewbogott: restarting each cloudcontrol galera node in turn to see if that quiets down the syncing warnings
* 12:08 arturo: move memory out of the swap in cloudcontrol1004 by disabling/enabling it (1Gb swap was being used)

=== 2020-12-22 ===
* 15:30 dcaro: cleaning up 6778 dangling snapshots for glance images in eqiad ([[phab:T270478|T270478]])
* 13:51 dcaro: merged patch to move wikidumpparse backups to cloudvirt1025 to free space on cloudvirt1026

=== 2020-12-19 ===
* 16:18 dcaro: gzipped a bunch of logs on cloudvirt1004 due to / being out of space
* 00:14 bstorm: truncated /var/log/debug.1 on cloudcontrol1003 which appears to be the exact same content as the user.log files anyway
* 00:10 bstorm: truncated /var/log/daemon.log.1 and the haproxy log
* 00:02 bstorm: truncated /var/log/messages.1 on cloudcontrol1003

=== 2020-12-18 ===
* 23:53 bstorm: truncated haproxy.log.1 on cloudcontrol1003
* 20:46 andrewbogott: setting  pg and pgp number to 4096 for eqiad1-compute as joachim thinks 8192 might be too much [[phab:T270305|T270305]]
* 17:09 dcaro: finished cleaning up the dangling snapshots from cloudvirt1026 ([[phab:T270478|T270478]])
* 17:08 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1026) ([[phab:T270478|T270478]])
* 17:06 dcaro: finished cleaning up the dangling snapshots from cloudvirt1025 ([[phab:T270478|T270478]])
* 17:05 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1025) ([[phab:T270478|T270478]])
* 17:00 dcaro: finished cleaning up the dangling snapshots from cloudvirt1021 ([[phab:T270478|T270478]])
* 16:58 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1021) ([[phab:T270478|T270478]])
* 16:56 dcaro: finished cleaning up the dangling snapshots from cloudvirt1022 ([[phab:T270478|T270478]])
* 16:55 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1022) ([[phab:T270478|T270478]])
* 16:54 dcaro: finished cleaning up the dangling snapshots from cloudvirt1023 ([[phab:T270478|T270478]])
* 16:51 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1023) ([[phab:T270478|T270478]])
* 16:47 dcaro: finished cleaning up the dangling snapshots from cloudvirt1024, freed ~12% of the capacity ([[phab:T270478|T270478]])
* 16:21 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1024) ([[phab:T270478|T270478]])
* 16:13 andrewbogott: setting  autoscale to 'off' for both ceph pools (eqiad1-compute and eqiad1-glance-images) because we like how things are set and the autoscaler does not
* 10:33 dcaro: purging rbd snapshots for image fc6fb78b-4515-4dcc-8254-{{Gerrit|591b9fe01762}} ([[phab:T270478|T270478]])

=== 2020-12-17 ===
* 22:17 andrewbogott: correction to above, set the pg and pgp to 1024 for eqiad1-glance-images
* 22:16 andrewbogott: setting  pgp number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) [[phab:T270305|T270305]] (same as pg)
* 22:14 andrewbogott: setting  pg number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) [[phab:T270305|T270305]]
* 22:10 andrewbogott: setting  autoscale to 'warn' for both ceph pools (eqiad1-compute and eqiad1-glance-images)

=== 2020-12-16 ===
* 09:31 dcaro: removing invalid backups from cloudvirt1024 (196 in total) ([[phab:T269419|T269419]])

=== 2020-12-14 ===
* 17:42 dcaro: The removal freed ~12GB (still 100% usage :S) ([[phab:T269419|T269419]])
* 17:36 dcaro: removing invalid backups that have a valid copy ([[phab:T269419|T269419]])
* 15:43 dcaro: Merging the tagging for vm backups ([[phab:T267195|T267195]])
* 09:45 arturo: icinga downtime cloudvirt1024 for 6 days ([[phab:T269419|T269419]])

=== 2020-12-13 ===
* 09:11 _dcaro: running backup purge script on cloudvirt1024 ([[phab:T269419|T269419]])

=== 2020-12-10 ===
* 23:36 bstorm: cleaned up the logs for haproxy on cloudcontrol1003 by deleting all the gzipped ones and truncating the .1 file
* 11:56 dcaro: Freed some space on cloudvirt1024 by running the purge script ([[phab:T269419|T269419]])
* 09:17 dcaro: removing leaked dns record discordwiki.eqiad.wmflabs (clinic duty)

=== 2020-12-08 ===
* 18:01 dcaro: Host cloudvirt1030 up and running ([[phab:T216195|T216195]])
* 15:59 dcaro: Re-imaging host cloudvirt1030 ([[phab:T216195|T216195]])
* 14:18 dcaro: Host online cloudvirt1029 ([[phab:T216195|T216195]])
* 14:13 dcaro: Host re-imaged, doing tests cloudvirt1029 ([[phab:T216195|T216195]])
* 12:14 dcaro: Re-imaging cloudvirt1029 ([[phab:T216195|T216195]])

=== 2020-12-07 ===
* 18:33 andrewbogott: putting cloudvirt1023 back into service [[phab:T269467|T269467]]
* 15:55 andrewbogott: reimaging cloudvirt1028 for [[phab:T216195|T216195]]
* 14:49 dcaro: Re-imaging cloudvirt1027 ([[phab:T216195|T216195]])

=== 2020-12-05 ===
* 00:35 andrewbogott: moving cloudvirt1023 back into maintenance because [[phab:T269467|T269467]] continues to puzzle

=== 2020-12-04 ===
* 22:33 andrewbogott: moving cloudvirt1023 back into the ceph aggregate; it doesn't need upgrades after all  [[phab:T269467|T269467]]
* 22:24 andrewbogott: moving cloudvirt1023 out of the ceph aggregate and into maintenance for [[phab:T269467|T269467]]
* 21:06 andrewbogott: putting cloudvirt1025 and 1026 back into service because I'm pretty sure they're fixed.  [[phab:T269313|T269313]]
* 12:12 arturo: manually running `wmcs-purge-backups` again on cloudvirt1024 ([[phab:T269419|T269419]])
* 11:25 arturo: icinga downtime cloudvirt1024 for 6 days, to avoid paging noises ([[phab:T269419|T269419]])
* 11:25 arturo: last log line referencing cloudvirt1024 is a mistake ([[phab:T269313|T269313]])
* 11:24 arturo: icinga downtime cloudvirt1024 for 6 days, to avoid paging noises ([[phab:T269313|T269313]])
* 10:28 arturo: manually running `wmcs-purge-backups` on cloudvirt1024 ([[phab:T269419|T269419]])
* 10:23 arturo: setting expiration to 2020-12-03 to the oldest backy snapshot of every VM in cloudvirt1024 ([[phab:T269419|T269419]])
* 09:54 arturo: icinga downtime cloudvirt1025 for 6 days ([[phab:T269313|T269313]])

=== 2020-12-03 ===
* 23:21 andrewbogott: removing all osds on cloudcephosd1004 for rebuild, [[phab:T268746|T268746]]
* 21:45 andrewbogott: removing all osds on cloudcephosd1005 for rebuild, [[phab:T268746|T268746]]
* 19:51 andrewbogott: removing all osds on cloudcephosd1006 for rebuild, [[phab:T268746|T268746]]
* 17:01 arturo: icinga downtime cloudvirt1025 for 48h to debug network issue [[phab:T269313|T269313]]
* 16:56 arturo: rebooting cloudvirt1025 to debug network issue [[phab:T269313|T269313]]
* 16:38 dcaro: Rimaging cloudvirt1026 ([[phab:T216195|T216195]])
* 13:24 andrewbogott: removing all osds on cloudcephosd1008 for rebuild, [[phab:T268746|T268746]]
* 02:55 andrewbogott: removing all osds on cloudcephosd1009 for rebuild, [[phab:T268746|T268746]]

=== 2020-12-02 ===
* 20:04 andrewbogott: removing all osds on cloudcephosd1010 for rebuild, [[phab:T268746|T268746]]
* 17:25 arturo: [15:51] failovering neutron virtual router in eqiad1 ([[phab:T268335|T268335]])
* 15:36 arturo: conntrackd is now up and running in cloudnet1003/1004 nodes ([[phab:T268335|T268335]])
* 15:33 arturo: [codfw1dev] conntrackd is now up and running in cloudnet200x-dev nodes ([[phab:T268335|T268335]])
* 15:08 andrewbogott: removing all osds on cloudcephosd1012 for rebuild, [[phab:T268746|T268746]]
* 12:41 arturo: disable puppet in all cloudnet servers to merge conntrackd change [[phab:T268335|T268335]]
* 11:12 dcaro: Reset the properties for the flavor g2.cores8.ram16.disk1120 to correct quotes ([[phab:T269172|T269172]])
* 09:57 arturo: moved cloudvirts 1030, 1029, 1028, 1027, 1026, 1025 away from the 'standard' host aggregate to 'maintenance' ([[phab:T269172|T269172]])

=== 2020-12-01 ===
* 20:06 andrewbogott: removing all osds on cloudcephosd1014 for rebuild, [[phab:T268746|T268746]]
* 12:04 arturo: restarting neutron l3 agents to pick up config change
* 11:48 arturo: merging change to dmz_dir, detail list of private address https://gerrit.wikimedia.org/r/c/operations/puppet/+/641977

=== 2020-11-30 ===
* 18:12 andrewbogott: removing all osds from cloudcephosd1015 in order to investigate [[phab:T268746|T268746]]

=== 2020-11-29 ===
* 17:18 andrewbogott: cleaning up some logfiles in tools-sgecron-01 — drive is full

=== 2020-11-26 ===
* 22:58 andrewbogott: deleting /var/log/haproxy logs older than 7 days in cloudcontrol100x.  We need log rotation here it seems.
* 15:53 dcaro: Created private flavor g2.cores8.ram16.disk1120 for wikidumpparse ([[phab:T268190|T268190]])

=== 2020-11-25 ===
* 19:35 bstorm: repairing ceph pg `instructing pg 6.91 on osd.117 to repair`
* 09:31 _dcaro: The OSD seems to be up and running actually, though there's that misleading log, will leave it see if the cluster comes fully healthy ([[phab:T268722|T268722]])
* 08:54 _dcaro: Unsetting noup/nodown to allow re-shuffling of the pgs that osd.44 had, will try to rebuild it ([[phab:T268722|T268722]])
* 08:45 _dcaro: Tried resetting the class for osd.44 to ssd, no luck, the cluster is in noout/norebalance to avoid data shuffling (opened [[phab:T268722|T268722]])
* 08:45 _dcaro: Tried resetting the class for osd.44 to ssd, no luck, the cluster is in noout/norebalance to avoid data shuffling (opened root@cloudcephosd1005:/var/lib/ceph/osd/ceph-44# ceph osd crush set-device-class ssd osd.44)
* 08:19 _dcaro: Restarting serivce osd.44 resulted on osd.44 being unable to start due to some config inconsistency (can not reset class to hdd)
* 08:16 _dcaro: After enabling auto pg scaling on ceph eqiad cluster, osd.44 (cloudcephosd1005) got stuck, trying to restart the osd service
* 08:16 _dcaro: After enabling auto pg scaling on ceph eqiad cluster, osd.44 (cloudcephosd1005) got stuck, trying to restart

=== 2020-11-22 ===
* 17:40 andrewbogott: apt-get upgrade on cloudservices1003/1004
* 17:32 andrewbogott: upgrading Designate on cloudservices1003/1004 to Stein

=== 2020-11-20 ===
* 12:44 arturo: [codfw1dev] install conntrackd in cloudnet2003-dev/cloudnet2002-dev to research l3 agent HA reliability
* 09:26 arturo: incinga downtime labstore1006 RAID checks for 10 days ([[phab:T268281|T268281]])

=== 2020-11-17 ===
* 19:21 andrewbogott: draining cloudvirt1012 to experiment with libvirt/cpu things

=== 2020-11-15 ===
* 11:21 arturo: icinga downtime cloudbackup2002 for 48h ([[phab:T267865|T267865]])

=== 2020-11-10 ===
* 16:38 arturo: icinga downtime toolschecker for 2h becasue toolsdb maintenance ([[phab:T266587|T266587]])
* 11:24 arturo: [codfw1dev] enable puppet in puppetmaster01.cloudinfra-codfw1dev (disabled for unspecified reasons)

=== 2020-11-09 ===
* 12:42 arturo: restarted neutron l3 agent in cloudnet1003 bc it still had the old default route ([[phab:T265288|T265288]])
* 12:41 arturo: `root@cloudcontrol1005:~# neutron subnet-delete dcbb0f98-5e9d-4a93-8dfc-4e3ec3c44dcc` ([[phab:T265288|T265288]])
* 12:41 arturo: `root@cloudcontrol1005:~# neutron router-gateway-set  --fixed-ip subnet_id=7c6bcc12-212f-44c2-9954-{{Gerrit|5c55002ee371}},ip_address=185.15.56.244 cloudinstances2b-gw wan-transport-eqiad` ([[phab:T265288|T265288]])
* 12:19 arturo: subnet 185.1.5.56.240/29 has id 7c6bcc12-212f-44c2-9954-{{Gerrit|5c55002ee371}} in neutron ([[phab:T265288|T265288]])
* 12:19 arturo: `root@cloudcontrol1005:~# neutron subnet-create --gateway 185.15.56.241 --name cloud-instances-transport1-b-eqiad1 --ip-version 4 --disable-dhcp wan-transport-eqiad 185.15.56.240/29` ([[phab:T265288|T265288]])
* 12:15 arturo: icinga-downtime toolschecker for 2h ([[phab:T265288|T265288]])

=== 2020-11-02 ===
* 13:36 arturo: (typo: dcaro)
* 13:35 arturo: added dcar as projectadmin &amp; user ([[phab:T266068|T266068]])

=== 2020-10-29 ===
* 16:57 bstorm: silenced deployment-prep project alerts for 60 days since the downtime expired
* 08:12 arturo: force-powercycling cloudcephosd1006

=== 2020-10-25 ===
* 16:20 andrewbogott: adding cloudvirt1038 to the 'ceph' aggregate and removing from the 'spare' aggregate.  We need this space while waiting on network upgrades for empty cloudvirts ([[phab:T216195|T216195]])

=== 2020-10-23 ===
* 11:30 arturo: [codfw1dev] openstack --os-project-id cloudinfra-codfw1dev recordset create --type PTR --record nat.cloudgw.codfw1dev.wikimediacloud.org. --description &quot;created by hand&quot; 0-29.57.15.185.in-addr.arpa. 1.0-29.57.15.185.in-addr.arpa. ([[phab:T261724|T261724]])
* 10:09 arturo: [codf1dev] doing DNS changes for the cloudgw PoC, including designate and https://gerrit.wikimedia.org/r/c/operations/dns/+/635965 ([[phab:T261724|T261724]])

=== 2020-10-22 ===
* 10:46 arturo: [codfw1dev] rebooting cloudinfra-internal-puppetmaster-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud to try fixing some DNS weirdness
* 09:43 arturo: enabling puppet in cloucontrol1003 (message said &quot;please re-enable after 2020-10-22 06:00UTC&quot;)

=== 2020-10-21 ===
* 14:36 andrewbogott: running apt-get update &amp;&amp; apt-get install -y facter on all cloud-vps instances
* 10:31 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])
* 08:56 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])

=== 2020-10-20 ===
* 15:47 arturo: changing DNS recursor ACLs (https://gerrit.wikimedia.org/r/c/operations/puppet/+/635314) this can be reverted any time if it causes problems ([[phab:T261724|T261724]])
* 14:49 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])

=== 2020-10-19 ===
* 01:41 andrewbogott: deleting all Precise base images
* 01:36 andrewbogott: deleting all unused Jessie base images

=== 2020-10-18 ===
* 23:26 andrewbogott: deleting all Trusty base images
* 21:50 andrewbogott: migrating all currently used ceph images to rbd

=== 2020-10-16 ===
* 09:29 arturo: [codfw1dev] still some DNS weirdness, investigating
* 09:25 arturo: [codfw1dev] hard-rebooting bastion-codfw1dev-02, seems in bad shape, doesn't even wake up in the virsh console
* 09:18 arturo: [codfw1dev] live-hacked cloudservices2002-dev /etc/powerdns/recursor.conf file to include cloud-codfw1dev-floating CIDR (185.15.57.0/29) while https://gerrit.wikimedia.org/r/c/operations/puppet/+/634050 is in review, so VMs with a floating IP can query the DNS recursor ([[phab:T261724|T261724]])
* 09:01 arturo: [codfw1dev] basic network connectivity seems stable after cleaning up everything related to address scopes ([[phab:T261724|T261724]])

=== 2020-10-15 ===
* 15:17 arturo: [codfw1dev] try cleaning up anything related to address scopes in the neutron database ([[phab:T261724|T261724]])
* 13:56 arturo: [codfw1dev] drop neutron l3 agent hacks in cloudnet2002/2003-dev ([[phab:T261724|T261724]])

=== 2020-10-13 ===
* 17:54 andrewbogott: rebuilding cloudvirt1021 for backy support
* 15:22 andrewbogott: draining cloudvirt1021 so I can rebuild it with backy support
* 14:19 andrewbogott: rebuilding cloudvirt1022 with backy support
* 14:03 andrewbogott: draining cloudvirt1022 so I can rebuild it with backy support
* 11:19 arturo: [codfw1dev] rebooting labtestvirt2003

=== 2020-10-09 ===
* 10:15 arturo: [codfwd1ev] root@cloudcontrol2001-dev:~# openstack router set --disable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw ([[phab:T261724|T261724]])
* 09:22 arturo: [codfwd1dev] rebooting cloudnet boxes for bridge and vlan changes ([[phab:T261724|T261724]])
* 09:12 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet delete 31214392-9ca5-4256-bff5-{{Gerrit|1e19a35661de}} (cloud-instances-transport1-b-codfw - 208.80.153.184/29) ([[phab:T261724|T261724]])
* 09:10 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack router set --external-gateway wan-transport-codfw --fixed-ip subnet=cloud-gw-transport-codfw,ip-address=185.15.57.10 cloudinstances2b-gw ([[phab:T261724|T261724]])
* 08:49 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.9 --no-dhcp --subnet-range 185.15.57.8/30 cloud-gw-transport-codfw ([[phab:T261724|T261724]])
* 08:47 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet delete a5ab5362-4ffb-4059-9ff7-{{Gerrit|391e22dcf3bc}} ([[phab:T261724|T261724]])

=== 2020-10-08 ===
* 16:17 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.8 --no-dhcp --subnet-range 185.15.57.8/31 cloud-gw-transport-codfw` (with a hack -- see task) ([[phab:T263622|T263622]])
* 16:03 arturo: [codfw1dev] briefly live-hacked python3-neutron source code in all 3 cloudcontrol2xxx-dev servers to workaround /31 network definition issue ([[phab:T263622|T263622]])
* 10:28 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) [[phab:T261724|T261724]]

=== 2020-10-06 ===
* 21:30 andrewbogott: moved cloudvirt1013 out of the 'ceph' aggregate and into the 'maintenance' aggregate for [[phab:T243414|T243414]]
* 21:29 andrewbogott: draining cloudvirt1013 for upgrade to 10G networking
* 14:45 arturo: icinga downtime every cloud* lab* host for 60 minutes for keystone maintenance

=== 2020-10-05 ===
* 17:40 bd808: `service uwsgi-labspuppetbackend restart` on cloud-puppetmaster-03 ([[phab:T264649|T264649]])

=== 2020-10-02 ===
* 11:05 arturo: [codfw1dev] restarting rabbitmq-server in all 3 control nodes, the l3 agent was misbehaving
* 09:16 arturo: [codfw1dev] trying the labtestvirt2003 (cloudgw) reimage again ([[phab:T261724|T261724]])

=== 2020-10-01 ===
* 16:06 arturo: rebooting cloudvirt1024 to validate changes to /etc/network/interfaces file
* 15:36 arturo: [codfw1dev] reimaging labtestvirt2003

=== 2020-09-30 ===
* 16:47 andrewbogott: rebooting cloudvir1032, 1033, 1034 for [[phab:T262979|T262979]]
* 13:28 arturo: enable puppet, reboot and pool back cloudvirt1031
* 13:27 arturo: extend icinga downtimes for another 120 mins
* 13:15 arturo: `aborrero@cloudcontrol1003:~$ sudo nova-manage placement sync_aggregates` after reading a hint in nova-api.log
* 13:02 arturo: rebooting cloudvirt1016 and moving it to the ceph host aggregate
* 12:55 arturo: rebooting cloudvirt1014 and moving it to the ceph host aggregate
* 12:51 arturo: rebooting cloudvirt1013 and moving it to the ceph host aggregate
* 12:39 arturo: root@cloudcontrol1005:~# openstack aggregate add host maintenance cloudvirt1031
* 12:36 arturo: rebooted cloudnet1003 (active) a couple of minutes ago
* 12:36 arturo: move cloudvirt1012 and cloudvirt1039 to the ceph aggregate
* 11:49 arturo: rebooting cloudvirt1039
* 11:46 arturo: rebooting cloudvirt1012
* 11:40 arturo: rebooting cloudnet1004 (standby) to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167 ([[phab:T262979|T262979]])
* 11:38 arturo: [codfw1dev] rebooting cloudnet2002-dev to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167
* 11:36 arturo: [codfw1dev] rebooting cloudnet2003-dev to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167
* 11:33 arturo: disabling puppet and downtiming every virt/net server in the fleet in preparation for merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167 ([[phab:T262979|T262979]])
* 09:32 arturo: rebooting cloudvirt1012 to investigate linuxbridge agent issues

=== 2020-09-29 ===
* 15:40 arturo: downgrade linux kernel from linux-image-4.19.0-11-amd64 to linux-image-4.19.0-10-amd64 on cloudvirt1012
* 14:47 arturo: rebooting cloudvirt1012, chasing config weirdness in the linuxbridge agent
* 14:05 andrewbogott: reimaging 1014 over and over in an attempt to get partman right
* 13:51 arturo: rebooting cloudvirt1012

=== 2020-09-28 ===
* 14:55 arturo: [jbond42] upgraded facter to v3 across the VM fleet
* 13:54 andrewbogott: moving cloudvirt1035 from aggregate 'spare' to 'ceph'.  We're going to need all the capacity we can get while converting older cloudvirts to ceph

=== 2020-09-24 ===
* 15:47 arturo: stopping/restarting rabbitmq-server in all cloudcontrol servers
* 15:45 arturo: restarting rabbitmq-server in cloudcontrol103
* 15:15 arturo: restarting floating_ip_ptr_records_updater.service in all 3 cloudcontrol servers to reset state after a DNS failure

=== 2020-09-18 ===
* 10:16 arturo: cloudvirt1039 libvirtd service issues were fixed with a reboot
* 09:56 arturo: rebooting cloudvirt1039 (spare) to try to fix some weird libvirtd failure
* 09:50 arturo: enabling puppet in cloudvirts and effectively merging patches from [[phab:T262979|T262979]]
* 08:59 arturo: disable puppet in all buster cloudvirts (cloudvirt[1024,1031-1039].eqiad.wmnet) to merge a patch for [[phab:T263205|T263205]] and [[phab:T262979|T262979]]
* 08:50 arturo: installing iptables from buster-bpo in cloudvirt1036 ([[phab:T263205|T263205]] and [[phab:T262979|T262979]])

=== 2020-09-15 ===
* 20:32 andrewbogott: rebooting cloudvirt1038 to see if it resolves [[phab:T262979|T262979]]
* 13:58 andrewbogott: draining cloudvirt1002 with wmcs-ceph-migrate

=== 2020-09-14 ===
* 14:21 andrewbogott: draining cloudvirt1001, migrating all VMs with wmcs-ceph-migrate
* 10:41 arturo: [codfw1dev] trying to get the bonding working for labtestvirt2003 ([[phab:T261724|T261724]])
* 09:47 arturo: installed qemu security update in eqiad1 cloudvirts ([[phab:T262386|T262386]])
* 09:43 arturo: [codfw1dev] installed qemu security update in codfw1dev cloudvirts ([[phab:T262386|T262386]])

=== 2020-09-09 ===
* 18:13 andrewbogott: restarting ceph-mon@cloudcephmon1003 in hopes that the slow ops reported are phantoms
* 18:01 andrewbogott: restarting ceph-mgr@cloudcephmon1003 in hopes that the slow ops reported are phantoms (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/EOWNO3MDYRUZKAK6RMQBQ5WBPQNLHOPV/)
* 17:40 andrewbogott: giving ceph pg autoscale another chance: ceph osd pool set eqiad1-compute pg_autoscale_mode on
* 00:05 bd808: Running wmcs-novastats-dnsleaks ([[phab:T262359|T262359]])

=== 2020-09-08 ===
* 21:48 bd808: Renamed FQDN prefixes to wikimedia.cloud scheme in cloudinfra-db01's labspuppet db ([[phab:T260614|T260614]])
* 14:29 andrewbogott: restarting nova-compute on all cloudvirts (everyone is upset from the reset switch failure)
* 14:18 arturo: restarting nova-fullstack service in cloudcontrol1003
* 14:17 andrewbogott: stopping apache2 on labweb1001 to make sure the Horizon outage is total

=== 2020-09-03 ===
* 09:31 arturo: icinga downtime cloud* servers for 30 mins ([[phab:T261866|T261866]])

=== 2020-09-02 ===
* 08:46 arturo: [codfw1dev] reimaging spare server labtestvirt2003 as debian buster ([[phab:T261724|T261724]])

=== 2020-09-01 ===
* 18:18 andrewbogott: adding drives on cloudcephosd100[3-5] to ceph osd pool
* 13:40 andrewbogott: adding drives on cloudcephosd101[0-2] to ceph osd pool
* 13:35 andrewbogott: adding drives on cloudcephosd100[1-3] to ceph osd pool
* 11:27 arturo: [codfw1dev] rebooting again cloudnet2002-dev after some network tests, to reset initial state ([[phab:T261724|T261724]])
* 11:09 arturo: [codfw1dev] rebooting cloudnet2002-dev after some network tests, to reset initial state ([[phab:T261724|T261724]])
* 10:49 arturo: disable puppet in cloudnet servers to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/623569/

=== 2020-08-31 ===
* 23:26 bd808: Removed stale lockfile at cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs:/var/lib/puppet/volatile/GeoIP/.geoipupdate.lock
* 11:20 arturo: [codfw1dev] livehacking https://gerrit.wikimedia.org/r/c/operations/puppet/+/615161 in the puppetmasters for tests before merging

=== 2020-08-28 ===
* 20:12 bd808: Running `wmcs-novastats-dnsleaks --delete` from cloudcontrol1003

=== 2020-08-26 ===
* 17:12 bstorm: Running 'ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf &quot;%k KB %p\n&quot; &gt; tools_large_files_20200826.txt' on labstore1004 [[phab:T261336|T261336]]

=== 2020-08-21 ===
* 21:34 andrewbogott: restarting nova-compute on cloudvirt1033; it seems stuck

=== 2020-08-19 ===
* 14:21 andrewbogott: rebooting cloudweb2001-dev, labweb1001, labweb1002 to address mediawiki-induced memleak

=== 2020-08-06 ===
* 21:02 andrewbogott: removing cloudvirt1004/1006 from nova's list of hypervisors; rebuilding them to  use as backup test hosts
* 20:06 bstorm: manually stopped the RAID check on cloudcontrol1003 [[phab:T259760|T259760]]

=== 2020-08-04 ===
* 18:54 bstorm: restarting mariadb on cloudcontrol1004 to setup parallel replication

=== 2020-08-03 ===
* 17:02 bstorm: increased db connection limit to 800 across galera cluster because we were clearly hovering at limit

=== 2020-07-31 ===
* 19:28 bd808: wmcs-novastats-dnsleaks --delete (lots of leaked fullstack-monitoring records to clean up)

=== 2020-07-27 ===
* 22:17 andrewbogott: ceph osd pool set compute pg_num 2048
* 22:14 andrewbogott: ceph osd pool set compute pg_autoscale_mode off

=== 2020-07-24 ===
* 19:15 andrewbogott: ceph mgr module enable pg_autoscaler
* 19:15 andrewbogott: ceph osd pool set compute pg_autoscale_mode on

=== 2020-07-22 ===
* 08:55 jbond42: [codfw1dev] upgrading hiera to version5
* 08:48 arturo: [codfw1dev] add jbond as user in the bastion-codfw1dev and cloudinfra-codfw1dev projects
* 08:45 arturo: [codfw1dev] enabled account creation in labtestwiki briefly for jbond42 to create an account

=== 2020-07-16 ===
* 10:48 arturo: merging change to neutron dmz_cidr https://gerrit.wikimedia.org/r/c/operations/puppet/+/613123 ([[phab:T257534|T257534]])

=== 2020-07-15 ===
* 23:15 bd808: Removed Merlijn van Deen from toollabs-trusted Gerrit group ([[phab:T255697|T255697]])
* 11:48 arturo: [codfw1dev] created DNS records (A and PTR) for bastion.bastioninfra-codfw1dev.codfw1dev.wmcloud.org &lt;-&gt; 185.15.57.2
* 11:41 arturo: [codfw1dev] add myself as projectadmin to the `bastioninfra-codfw1dev` project
* 11:39 arturo: [codfw1dev] created DNS zone `bastioninfra-codfw1dev.codfw1dev.wmcloud.org.` in the cloudinfra-codfw1dev project and then transfer ownership to the bastioninfra-codfw1dev project

=== 2020-07-14 ===
* 15:19 arturo: briefly set root@cloudnet1003:~ # sysctl net.ipv4.conf.all.accept_local=1 (in neutron qrouter netns) ([[phab:T257534|T257534]])
* 10:43 arturo: icinga downtime cloudnet* hosts for 30 mins to introduce new check https://gerrit.wikimedia.org/r/c/operations/puppet/+/612390 ([[phab:T257552|T257552]])
* 04:01 andrewbogott: added a wildcard *.wmflabs.org domain pointing at the domain proxy in project-proxy
* 04:00 andrewbogott: shortened the ttl on .wmflabs.org. to 300

=== 2020-07-13 ===
* 16:17 arturo: icinga downtime cloudcontrol[1003-1005].wikimedia.org for 1h for galera database movements

=== 2020-07-12 ===
* 17:39 andrewbogott: switched eqiad1 keystone from m5 to cloudcontrol galera

=== 2020-07-10 ===
* 20:26 andrewbogott: disabling nova api to move database to galera

=== 2020-07-09 ===
* 11:23 arturo: [codfw1dev] rebooting cloudnet2003-dev again for testing sysct/puppet behavior ([[phab:T257552|T257552]])
* 11:11 arturo: [codfw1dev] rebooting cloudnet2003-dev for testing sysct/puppet behavior ([[phab:T257552|T257552]])
* 09:16 arturo: manually increasing sysctl value of net.nf_conntrack_max in cloudnet servers ([[phab:T257552|T257552]])

=== 2020-07-06 ===
* 15:16 arturo: installing 'aptitude' in all cloudvirts

=== 2020-07-03 ===
* 12:51 arturo: [codfw1dev] galera cluster should be up and running, openstack happy ([[phab:T256283|T256283]])
* 11:44 arturo: [codfw1dev] restoring glance database backup from bacula into cloudcontrol2001-dev ([[phab:T256283|T256283]])
* 11:39 arturo: [codfw1dev] stopped mysql database in the galera cluster [[phab:T256283|T256283]]
* 11:36 arturo: [codfw1dev] dropped glance database in the galera cluster [[phab:T256283|T256283]]

=== 2020-07-02 ===
* 15:41 arturo: `sudo wmcs-openstack --os-compute-api-version 2.55 flavor create --private --vcpus 8 --disk 300 --ram 16384 --property aggregate_instance_extra_specs:ceph=true --description &quot;for packaging envoy&quot; bigdisk-ceph` ([[phab:T256983|T256983]])

=== 2020-06-29 ===
* 14:24 arturo: starting rabbitmq-server in all 3 cloudcontrol servers
* 14:23 arturo: stopping rabbitmq-server in all 3 cloudcontrol servers

=== 2020-06-18 ===
* 20:38 andrewbogott: rebooting cloudservices2003-dev due to a mysterious 'host down' alert on a secondary ip

=== 2020-06-16 ===
* 15:38 arturo: created by hand neutron port 9c0a9a13-e409-49de-9ba3-{{Gerrit|bc8ec4801dbf}} `paws-haproxy-vip` ([[phab:T295217|T295217]])

=== 2020-06-12 ===
* 13:23 arturo: DNS zone `paws.wmcloud.org` transferred to the PAWS project ([[phab:T195217|T195217]])
* 13:20 arturo: created DNS zone `paws.wmcloud.org` ([[phab:T195217|T195217]])

=== 2020-06-11 ===
* 19:19 bstorm_: proceeding with failback to labstore1004 now that DRBD devices are consistent [[phab:T224582|T224582]]
* 17:22 bstorm_: delaying failback labstore1004 for drive syncs [[phab:T224582|T224582]]
* 17:17 bstorm_: failing NFS back to labstore1004 to complete the upgrade process [[phab:T224582|T224582]]
* 16:15 bstorm_: failing over NFS for labstore1004 to labstore1005 [[phab:T224582|T224582]]

=== 2020-06-10 ===
* 16:09 andrewbogott: deleting all old cloud-ns0.wikimedia.org and cloud-ns1.wikimedia.org ns records in designate database [[phab:T254496|T254496]]

=== 2020-06-09 ===
* 15:25 arturo: icinga downtime everything cloud* lab* for 2h more ([[phab:T253780|T253780]])
* 14:09 andrewbogott: stopping puppet, all designate services and all pdns services on cloudservices1004 for [[phab:T253780|T253780]]
* 14:01 arturo: icinga downtime everything cloud* lab* for 2h ([[phab:T253780|T253780]])

=== 2020-06-05 ===
* 15:08 andrewbogott: trying to re-enable puppet without losing cumin contact, as per https://phabricator.wikimedia.org/T254589

=== 2020-06-04 ===
* 14:24 andrewbogott: disabling puppet on all instances for /labs/private recovery
* 14:23 arturo: disabling puppet on all instances for /labs/private recovery

=== 2020-05-28 ===
* 23:02 bd808: `/usr/local/sbin/maintain-dbusers --debug harvest-replicas` ([[phab:T253930|T253930]])
* 13:36 andrewbogott: rebuilding cloudservices2002-dev with Buster
* 00:33 andrewbogott: shutting down cloudservices2002-dev to see if we can live without it.  This is in anticipation or rebuilding it entirely for [[phab:T253780|T253780]]

=== 2020-05-27 ===
* 23:29 andrewbogott: disabling the backup job on cloudbackup2001 (just like last week) so the backup doesn't start while Brooke is rebuilding labstore1004 tomorrow.
* 06:03 bd808: `systemctl start mariadb` on clouddb1001 following reboot (take 2)
* 05:58 bd808: `systemctl start mariadb` on clouddb1001 following reboot
* 05:53 bd808: Hard reboot of clouddb1001 via Horizon. Console unresponsive.

=== 2020-05-25 ===
* 16:35 arturo: [codfw1dev] created zone `0-29.57.15.185.in-addr.arpa.` ([[phab:T247972|T247972]])

=== 2020-05-21 ===
* 19:23 andrewbogott: disabling puppet on cloudbackup2001 to prevent the backup job from starting during maintenance
* 19:16 andrewbogott: systemctl disable block_sync-tools-project.service on cloudbackup2001.codfw.wmnet to avoid stepping on current upgrade
* 15:48 andrewbogott: re-imaging cloudnet1003 with Buster

=== 2020-05-19 ===
* 22:59 bd808: `apt-get install mariadb-client` on cloudcontrol1003
* 21:12 bd808: Migrating wcdo.wcdo.eqiad.wmflabs to cloudvirt1023 ([[phab:T251065|T251065]])

=== 2020-05-18 ===
* 21:37 andrewbogott: rebuilding cloudnet2003-dev with Buster

=== 2020-05-15 ===
* 22:10 bd808: Added reedy as projectadmin in cloudinfra project ([[phab:T249774|T249774]])
* 22:05 bd808: Added reedy as projectadmin in admin project ([[phab:T249774|T249774]])
* 18:44 bstorm_: rebooting cloudvirt-wdqs1003 [[phab:T252831|T252831]]
* 15:47 bd808: Manually running wmcs-novastats-dnsleaks from cloudcontrol1003 ([[phab:T252889|T252889]])

=== 2020-05-14 ===
* 23:28 bstorm_: downtimed cloudvirt1004/6 and cloudvirt-wdqs1003 until tomorrow around this time [[phab:T252831|T252831]]
* 22:21 bstorm_: upgrading qemu-system-x86 on cloudvirt1006 to backports version [[phab:T252831|T252831]]
* 22:15 bstorm_: changing /etc/libvirt/qemu.conf and restarting libvirtd on cloudvirt1006 [[phab:T252831|T252831]]
* 21:12 andrewbogott: rebuilding cloudvirt1003-wdqs as part of [[phab:T252831|T252831]]
* 15:47 andrewbogott: moving cloudvirt1004 and cloudvirt1006 to the 'ceph' aggregate for [[phab:T252784|T252784]]
* 15:02 andrewbogott: moving all of cloudvirt100[1-9] into the 'toobusy' host aggregate.  These are slower, have spinning disks, and are due for replacement.

=== 2020-05-12 ===
* 20:33 andrewbogott: moving cloudvirt1023 to the 'standard' pool and out of the 'spare' pool
* 19:10 jeh: disable neutron-openvswitch-agent service on cloudvirt2001-dev.codfw [[phab:T248881|T248881]]
* 19:09 jeh: Shutdown the unused eno2 network interface on cloudvirt2001-dev.codfw to clear up monitoring errors [[phab:T248425|T248425]]
* 18:20 andrewbogott: moving cloudvirt1024 out of the 'maintenance' aggregate and into 'spare'
* 16:45 andrewbogott: restarting neutron-l3-agent on cloudnet1004 so it knows about all three cloudcontrols.  Leaving cloudnet1003 since restarting it there will cause network interruptions
* 14:06 arturo: icinga downtime everything for 2h for Debian Buster migration in some cloud components

=== 2020-05-09 ===
* 16:53 andrewbogott: rebuilding cloudcontrol2001-dev and 2003-dev with buster for [[phab:T252121|T252121]]

=== 2020-05-08 ===
* 19:02 bstorm_: moving tools-k8s-haproxy-2 from cloudvirt1021 to cloudvirt1017 to improve spread

=== 2020-05-05 ===
* 13:58 andrewbogott: rebuilding cloudcontrol2004-dev to test new puppet changes

=== 2020-05-04 ===
* 09:04 arturo: [codfw1dev] manually modify iptables ruleset to only allow SSH from WMF bastions on cloudservices2003-dev and cloudcontrol2004-dev ([[phab:T251604|T251604]])

=== 2020-04-21 ===
* 22:12 andrewbogott: moving cloudvirt1004 out of the 'standard' aggregate and into the 'maintenance' aggregate
* 16:01 jeh: restart cloudceph mon and osd services for openssl upgrades

=== 2020-04-15 ===
* 18:44 jeh: create indexes and views for grwikimedia [[phab:T245912|T245912]]

=== 2020-04-13 ===
* 15:07 jeh: restart memcached on labwebs to increase cache size [[phab:T145703|T145703]]

=== 2020-04-09 ===
* 19:57 andrewbogott: upgrading eqiad1 designate to rocky
* 16:52 andrewbogott: cleaned up a bunch of leaked .eqiad.wmflabs dns records

=== 2020-04-08 ===
* 19:20 andrewbogott: rotated password and api token for pdns servers on cloudservices1003 and cloudservices1004
* 14:54 arturo: `root@cloudcontrol1003:~# cp /etc/inputrc .inputrc` to solve some bash shortcut weirdness

=== 2020-04-07 ===
* 20:57 andrewbogott: service sssd stop; rm -rf /var/lib/sss/db*; service sssd start on tools-sgebastion-08

=== 2020-04-06 ===
* 22:39 andrewbogott: deleting bogus groups cn=b'project-bastion',ou=groups,dc=wikimedia,dc=org and cn=b'project-tools',ou=groups,dc=wikimedia,dc=org from ldap
* 17:42 arturo: [codfw1dev] transferred DNS zone 57.15.185.in-addr.arpa. to the cloudinfra-codfw1dev project ([[phab:T247972|T247972]])
* 17:39 arturo: [codfw1dev] `openstack zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description &quot;floating IPs subnet&quot; 57.15.185.in-addr.arpa.` ([[phab:T247972|T247972]])
* 16:23 arturo: restarting apache2 in cloudcontrol1003/1004 to pick up latest wmfkeystonehooks changes [[phab:T249494|T249494]]

=== 2020-04-02 ===
* 20:59 jeh: codfw1dev clear VM error states and start bastions, puppet master and database

=== 2020-04-01 ===
* 16:27 arturo: [codfw1dev] enable puppet across the fleet clean vxlan changes ([[phab:T248881|T248881]])

=== 2020-03-31 ===
* 12:35 arturo: [codfw1dev] restarting VMs: designaterockytest14, bastion-codfw1dev-0[1,2] ([[phab:T248881|T248881]])
* 12:34 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudvirt2001-dev ([[phab:T248881|T248881]])
* 12:25 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudnet200[2,3]-dev ([[phab:T248881|T248881]])
* 11:45 arturo: [codfw1dev] rebooting cloudvirt2003-dev to pick up latest kernel update. Otherwise modprobe is confused trying to load modules and openvswitch won't start ([[phab:T248881|T248881]])
* 10:40 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudvirt2003-dev ([[phab:T248881|T248881]])
* 10:09 arturo: [codfw1dev] reboot cloudnet2003-dev into linux 4.9 (was using 4.14 from a testing operation in 2020-03-10)

=== 2020-03-30 ===
* 23:42 bstorm_: deleted &quot;Kubernetes Cluster&quot; and &quot;Kubernetes Performance&quot; dashboards [[phab:T246689|T246689]]
* 16:44 arturo: [codfw1dev] installing package neutron-openvswitch-agent in cloudvirt2002-dev ([[phab:T248881|T248881]])
* 16:42 andrewbogott: restarting l3 agents on cloudnets in codfw1dev after applying https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/584188/

=== 2020-03-27 ===
* 21:28 bd808: Created huggle.wmcloud.org Designate zone and allocated it to the huggle project
* 19:51 jeh: start haproxy on cloudcontrol2003-dev.wikimedia.org

=== 2020-03-26 ===
* 15:01 arturo: icinga downtime cloudvirt* cloudcontrol* cloudnet* lab* cloudstore*
* 15:01 andrewbogott: beginning openstack upgrade window for [[phab:T242766|T242766]]
* 12:32 arturo: [codfw1dev] downgraded systemd, libsystemd0, udev and friends to the non-backports versions ([[phab:T247013|T247013]])

=== 2020-03-25 ===
* 19:29 andrewbogott: dumping a bunch of VMs on cloudvirt1015 to see if it still crashes
* 17:56 jeh: add labweb1002 back into the pool - completed horizon testing [[phab:T240852|T240852]]
* 17:09 jeh: depool labweb1002 for horizon testing [[phab:T240852|T240852]]

=== 2020-03-24 ===
* 19:41 jeh: switch cloudvirt1016 from maintenance to standard host aggregate [[phab:T243327|T243327]]
* 15:31 andrewbogott: restarting nova-conductor and nova-api on cloudcontrol1003 and cloudcontrol1004

=== 2020-03-23 ===
* 21:41 jeh: restart neutron-l3-agent on cloudnet100[3,4] to pickup policy.yaml changes
* 13:28 jeh: disable puppet on labweb100[1,2] to enable horizon event traces [[phab:T240852|T240852]]
* 10:26 arturo: restarting apache in both labweb1001/labweb1002 upon reports of returning 500s

=== 2020-03-21 ===
* 14:23 andrewbogott: restarting apache2 on labweb1001 and 1002

=== 2020-03-18 ===
* 19:17 andrewbogott: deleted a bunch of records from the pdns database on cloudservices1003/1004 which had a record name but the content (where an IP address should be) was NULL, e.g. m.wikidata.beta.wmflabs.org.
* 10:55 arturo: [codfw1dev] deleting BGP agent, undoing changes we did for [[phab:T245606|T245606]]

=== 2020-03-14 ===
* 17:40 jeh: restart maintain-dbusers on labstore1004 [[phab:T247654|T247654]]

=== 2020-03-13 ===
* 12:39 arturo: [codfw1dev] reintroduce address scopes for another round of testing [[phab:T244851|T244851]]
* 12:17 arturo: [codfw1dev] enabling puppet in cloudnet200x-dev servers after merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/579259 ([[phab:T247505|T247505]])

=== 2020-03-12 ===
* 22:29 bstorm_: running puppet across all dumps mounts to make sure active links are shifted to labstore1006

=== 2020-03-11 ===
* 18:38 jeh: set icingia downtime until 2020-03-23 on CODFW cloud[control,net,virt] hosts during openstack upgrades
* 12:50 arturo: [codfw1dev] several tests creating/deleting address scopes ([[phab:T244727|T244727]] [[phab:T247135|T247135]] [[phab:T246887|T246887]] [[phab:T245606|T245606]])
* 12:46 arturo: [codfw1dev] disable routing_source_ip in l3 agents for testing proposal detailed at https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Network_refresh#Eliminate_routing_source_ip_address ([[phab:T244727|T244727]])

=== 2020-03-10 ===
* 17:02 arturo: [codfw1dev] deleting address scopes, bad interaction with our custom NAT setup [[phab:T247135|T247135]]
* 13:55 arturo: [codfw1dev] rebooting cloudnet2003-dev into linux kernel 4.14 for testing stuff related to [[phab:T247135|T247135]]

=== 2020-03-09 ===
* 18:09 arturo: enabling puppet in cloudvirt1006, all services have been restored
* 17:59 arturo: deleted the neutron bridge on cloudvirt1006, for testing stuff related to the queens upgrade
* 17:58 arturo: stopped neutron-linuxbridge-agent and nova-compute in cloudvirt1006 for testing stuff related to the queens upgrade

=== 2020-03-06 ===
* 14:54 andrewbogott: draining all instances off of cloudvirt1006 for [[phab:T246908|T246908]]

=== 2020-03-05 ===
* 14:24 arturo: [codfw1dev] we just enabled BGP session between cloudnet2xxx-dev and cr1-codfw ([[phab:T245606|T245606]])
* 13:07 arturo: [codfw1dev] move the extra IP address for BGP in cloudnet200x-dev servers from eno2.2120 to the br-external bridge device ([[phab:T245606|T245606]])
* 13:06 arturo: [codfw1dev] upgrade neutron-dynamic-routing packages in cloudnet200X-dev and cloudcontrol200X-dev servers to 11.0.0-2~bpo9+1 ([[phab:T245606|T245606]])

=== 2020-03-04 ===
* 22:22 andrewbogott: upgrading designate on cloudservices1003/1004 to Queens
* 22:09 andrewbogott: moving cloudvirt1006 into the maintenance aggregate for [[phab:T246908|T246908]]
* 21:37 bd808: Running wmcs-wikireplica-dns to add service names for ngwikimedia.*.db.svc.eqiad.wmflabs ([[phab:T240772|T240772]])
* 21:14 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1009 ([[phab:T246056|T246056]])
* 21:11 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1010 ([[phab:T246056|T246056]])
* 21:08 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1011 ([[phab:T246056|T246056]])
* 21:05 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1002 ([[phab:T246056|T246056]])

=== 2020-03-02 ===
* 16:54 arturo: [codfw1dev] deleted python3-os-ken debian package in cloudnet2003-dev which was installed by hand and had depedency issues

=== 2020-02-29 ===
* 16:32 bstorm_: downtimed the smart alert on cloudvirt1009 until Monday since apparently predictive failures flap [[phab:T244986|T244986]]

=== 2020-02-26 ===
* 22:03 jeh: powering down cloudvirt1014 for hardware maintenance

=== 2020-02-25 ===
* 16:08 andrewbogott: changing neutron's rabbitmq password because oslo is having trouble parsing some of the characters in the password
* 15:26 andrewbogott: updated the cell_mapping record in the nova_api database to add the second rabbitmq server  to the transport_url field
* 15:26 andrewbogott: updated the cell_mapping record in the nova_api database to set the db uri to 'mysql+pymysql' -- this in response to a deprecation notice

=== 2020-02-24 ===
* 12:16 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker cr2-codfw` ([[phab:T245606|T245606]])
* 12:16 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker cr1-codfw` ([[phab:T245606|T245606]])
* 12:09 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.187 --remote-as 65002 cr2-codfw` ([[phab:T245606|T245606]])
* 12:09 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.186 --remote-as 65002 cr1-codfw` ([[phab:T245606|T245606]])
* 12:06 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-delete 17b8c2a3-f0ce-4d50-a265-18ccac703c61` ([[phab:T245606|T245606]])
* 10:59 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker bgppeer` ([[phab:T245606|T245606]])
* 10:56 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.185 --remote-as 65002 bgppeer` ([[phab:T245606|T245606]])

=== 2020-02-21 ===
* 12:48 arturo: [codfw1dev] running `root@cloudcontrol2001-dev:~# neutron bgp-speaker-network-add bgpspeaker wan-transport-codfw` ([[phab:T245606|T245606]])
* 12:46 arturo: [codfw1dev] created bgpspeaker for AS64711 ([[phab:T245606|T245606]])
* 12:42 arturo: [codfw1dev] run `sudo neutron-db-manage upgrade head` to upgrade the db schema for neutron bgp tables
* 11:51 arturo: [codfw1dev] create a neutron subnet pool per each subnet objects we have and manually update DB to inter-associate them ([[phab:T245606|T245606]])
* 11:49 arturo: [codfw1dev] rename neutron address scope `no-nat` to `bgp` ([[phab:T245606|T245606]])
* 11:37 arturo: [codfw1dev] cleanup unused neutron subnet pools from previous address scope tests ([[phab:T244851|T244851]])

=== 2020-02-20 ===
* 19:22 andrewbogott: updating designate pool config for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/572213/
* 15:33 andrewbogott: migrating all VMs on cloudvirt1014 to cloudvirt1022
* 13:35 arturo: [codfw1dev] disable puppet in cloudcontrol servers to hack neutron.conf for tests related to [[phab:T245606|T245606]]
* 13:33 arturo: [codfw1dev] disable puppet in cloudnet servers to hack neutron.conf for tests related to [[phab:T245606|T245606]]

=== 2020-02-18 ===
* 22:19 andrewbogott: transferred the tools.wmcloud.org. to the tools project
* 22:16 andrewbogott: moved wmcloud.org dns domain to the cloud-infra project
* 21:02 andrewbogott: adding .eqiad1.wikimedia.cloud records to all existing eqiad1 VMs, updating all eqiad1 internal pointer records to reference the new eqiad1.wikimedia.cloud fqdns.
* 09:44 arturo: deleted DNS zone wmcloud.org and try re-creating it

=== 2020-02-14 ===
* 10:35 arturo: running `root@cloudcontrol2001-dev:~# designate server-create --name ns1.openstack.codfw1dev.wikimediacloud.org.` ([[phab:T243766|T243766]])
* 10:32 arturo: running `root@cloudcontrol1004:~# designate server-create --name ns1.openstack.eqiad1.wikimediacloud.org.` ([[phab:T243766|T243766]])
* 10:32 arturo: running `root@cloudcontrol1004:~# designate server-create --name ns0.openstack.eqiad1.wikimediacloud.org.` ([[phab:T243766|T243766]])

=== 2020-02-12 ===
* 13:38 arturo: [codfw1dev] add reference to subnetpool to the instance subnet `MariaDB [neutron]&gt; update subnets set subnetpool_id='d129650d-d4be-4fe1-b13e-6edb5565cb4a' where id = '7adfcebe-b3d0-4315-92fe-e8365cc80668';` ([[phab:T244851|T244851]])

=== 2020-02-11 ===
* 13:46 arturo: [codfw1dev] creating some neutron objects to investigate [[phab:T244851|T244851]] (subnets, subnet pools, address scopes, ...)
* 12:40 arturo: [codfw1dev] delete unknown address scope 'wmcs-v4-scope': `root@cloudcontrol2001-dev:~# openstack address scope delete 078cfd71-117b-4aac-9197-6ebbbb7dd3de` ([[phab:T244851|T244851]])
* 12:40 arturo: [codfw1dev] delete unknown subnet pool 'cloudinstancesb-v4-pool0': `root@cloudcontrol2001-dev:~# openstack subnet pool delete d23a9b88-5c3d-4a53-ab88-053233a75365` ([[phab:T244851|T244851]])

=== 2020-02-07 ===
* 18:11 jeh: shutdown cloudvirt1016 for hardware maintenance [[phab:T241882|T241882]]

=== 2020-02-06 ===
* 14:44 jeh: update apt packages on cloudvirt1015 [[phab:T220853|T220853]]
* 14:28 jeh: run hardware tests on cloudvirt1015 [[phab:T220853|T220853]]

=== 2020-01-28 ===
* 17:24 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# designate server-create --name ns0.openstack.codfw1dev.wikimediacloud.org. ([[phab:T243766|T243766]])
* 10:18 arturo: [codfw1dev] created DNS record `bastion-codfw1dev-01.codfw1dev.wmcloud.org A 185.15.57.2` ([[phab:T242976|T242976]], [[phab:T229441|T229441]])
* 10:13 arturo: [codfw1dev] the zone `codfw1dev.wmcloud.org` belongs now to the `cloudinfra-codfw1dev` project ([[phab:T242976|T242976]])
* 10:11 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description &quot;main DNS domain for public addresses&quot; --email &quot;root@wmflabs.org&quot; --type PRIMARY --ttl 3600 codfw1dev.wmcloud.org.` ([[phab:T242976|T242976]] and [[phab:T243766|T243766]])
* 09:53 arturo: restart apache2 in labweb1001/1002 because horizon errors
* 09:47 arturo: created DNS zone wmcloud.org in eqiad1, transfer it to the cloudinfra project ([[phab:T242976|T242976]]) right now only use is to delegate codfw1dev.wmcloud.org subdomain to designate in the other deployment

=== 2020-01-27 ===
* 12:45 arturo: [codfw1dev] manually move the new domain to the `cloudinfra-codfw1dev` project clouddb2001-dev: `[designate]&gt; update zones set tenant_id='cloudinfra-codfw1dev' where id = '4c75410017904858a5839de93c9e8b3d';` [[phab:T243556|T243556]]
* 12:44 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description &quot;main DNS domain for VMs&quot; --email &quot;root@wmflabs.org&quot; --type PRIMARY --ttl 3600 codfw1dev.wikimedia.cloud.` [[phab:T243556|T243556]]

=== 2020-01-24 ===
* 15:10 jeh: remove icinga downtime for cloudvirt1013 [[phab:T241313|T241313]]
* 12:52 arturo: repooling cloudvirt1013 after HW got fixed ([[phab:T241313|T241313]])

=== 2020-01-21 ===
* 17:43 bstorm_: remounting /mnt/nfs/dumps-labstore1007.wikimedia.org/ on all dumps-mounting projects
* 10:24 arturo: running `sudo systemctl restart apache2.service` in both labweb servers to try mitigating [[phab:T240852|T240852]]

=== 2020-01-15 ===
* 16:59 bd808: Changed the config for cloud-announce mailing list so that lsit admins do not get bounce unsubscribe notices

=== 2020-01-14 ===
* 14:03 arturo: icinga downtime all cloudvirts for another 2h for fixing some icinga checks
* 12:04 arturo: icinga downtime toolchecker for 2 hours for openstack upgrades [[phab:T241347|T241347]]
* 12:02 arturo: icinga downtime cloud* labs* hosts for 2 hours for openstack upgrades [[phab:T241347|T241347]]
* 04:26 andrewbogott: upgrading designate on cloudservices1003/1004

=== 2020-01-13 ===
* 13:34 arturo: [¢odfw1dev] prevent neutron from allocating floating IPs from the wrong subnet by doing `neutron subnet-update --allocation-pool start=208.80.153.190,end=208.80.153.190 cloud-instances-transport1-b-codfw` ([[phab:T242594|T242594]])

=== 2020-01-10 ===
* 13:27 arturo: cloudvirt1009: virsh undefine i-000069b6. This is tools-elastic-01 which is running on cloudvirt1008 (so, leaked on cloudvirt1009)

=== 2020-01-09 ===
* 11:12 arturo: running `MariaDB [nova_eqiad1]&gt; update quota_usages set in_use='0' where project_id='etytree';` ([[phab:T242332|T242332]])
* 11:11 arturo: running `MariaDB [nova_eqiad1]&gt; select * from quota_usages where project_id = 'etytree';` ([[phab:T242332|T242332]])
* 10:32 arturo: ran `root@cloudcontrol1004:~# nova-manage project quota_usage_refresh --project etytree`

=== 2020-01-08 ===
* 10:53 arturo: icinga downtime all cloudvirts for 30 minutes to re-create all canary VMs&quot;

=== 2020-01-07 ===
* 11:12 arturo: icinga-downtime everything cloud* for 30 minutes to merge nova scheduler changes
* 10:02 arturo: icinga downtime cloudvirt1009 for 30 minutes to re-create canary VM ([[phab:T242078|T242078]])

=== 2020-01-06 ===
* 13:45 andrewbogott: restarting nova-api and nova-conductor on cloudcontrol1003 and 1004

=== 2020-01-04 ===
* 16:34 arturo: icinga downtime cloudvirt1024 for 2 months because hardware errors ([[phab:T241884|T241884]])

=== 2019-12-31 ===
* 11:46 andrewbogott: I couldn't!
* 11:40 andrewbogott: restarting cloudservices2002-dev to see if I can reproduce an issue I saw earlier

=== 2019-12-25 ===
* 10:13 arturo: icinga downtime for 30 minutes the whole cloud* lab* fleet to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/560575 (will restart some openstack components)

=== 2019-12-24 ===
* 15:13 arturo: icinga downtime all the lab* fleet for nova password change for 1h
* 14:39 arturo: icinga downtime all the cloud* fleet for nova password change for 1h

=== 2019-12-23 ===
* 11:13 arturo: enable puppet in cloudcontrol1003/1004
* 10:40 arturo: disable puppet in cloudcontrol1003/1004 while doing changes related to python-ldap

=== 2019-12-22 ===
* 23:48 andrewbogott: restarting nova-conductor and nova-api on cloudcontrol1003 and 1004
* 09:45 arturo: cloudvirt1013 is back (did it alone) [[phab:T241313|T241313]]
* 09:37 arturo: cloudvirt1013 is down for good. Apparently powered off. I can't even reach it via iLO

=== 2019-12-20 ===
* 12:43 arturo: icinga downtime cloudmetrics1001 for 128 hours

=== 2019-12-18 ===
* 12:55 arturo: [codfw1dev] created a new subnet neutron object to hold the new CIDR for floating IPs (cloud-codfw1dev-floating - 185.15.57.0/29) [[phab:T239347|T239347]]

=== 2019-12-17 ===
* 07:21 andrewbogott: deploying horizon/train to labweb1001/1002

=== 2019-12-12 ===
* 06:11 arturo: schedule 4h downtime for labstores
* 05:57 arturo: schedule 4h downtime for cloudvirts and other openstack components due to upgrade ops

=== 2019-12-02 ===
* 06:28 andrewbogott: running  nova-manage db sync on eqiad1
* 06:27 andrewbogott: running nova-manage cell_v2 map_cell0 on eqiad1

=== 2019-11-21 ===
* 16:07 jeh: created replica indexes and views for szywiki [[phab:T237373|T237373]]
* 15:48 jeh: creating replica indexes and views for shywiktionary [[phab:T238115|T238115]]
* 15:48 jeh: creating replica indexes and views for gcrwiki [[phab:T238114|T238114]]
* 15:46 jeh: creating replica indexes and views for minwiktionary [[phab:T238522|T238522]]
* 15:36 jeh: creating replica indexes and views for gewikimedia [[phab:T236404|T236404]]

=== 2019-11-18 ===
* 19:27 andrewbogott: repooling labsdb1011
* 18:54 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1011    [[phab:T238480|T238480]]
* 18:44 andrewbogott: depooling labsdb1011 and killing remaining user queries  [[phab:T238480|T238480]]
* 18:42 andrewbogott: repooled labsdb1009 and 1010  [[phab:T238480|T238480]]
* 18:19 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1010    [[phab:T238480|T238480]]
* 18:18 andrewbogott: depooling labsdb1010, killing remaining user queries
* 17:46 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1009    [[phab:T238480|T238480]]
* 17:38 andrewbogott: depooling labsdb1009, killing remaining user queries
* 16:54 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1012   [[phab:T237509|T237509]]

=== 2019-11-15 ===
* 20:04 andrewbogott: repool labdb1011 ([[phab:T237509|T237509]])
* 19:29 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1011
* 19:25 andrewbogott: depooling labsdb1011, killing remaining queries
* 19:25 andrewbogott: repooling labsdb1010
* 18:59 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1012
* 18:57 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1010
* 18:54 andrewbogott: depooling labsdb1010, killing remaining user queries
* 18:54 andrewbogott: depooled labsdb1009, ran maintain-views —clean —all-databases —replace-all, repooled

=== 2019-11-11 ===
* 13:10 arturo: cloudweb2001-dev: disable puppet and redirect stderr in the loadExitNodes.php cron script to prevent cronspam while we investigate the cause of the issue ([[phab:T237971|T237971]])

=== 2019-11-05 ===
* 11:59 arturo: icinga downtime for 1h cloudcontrol1004, cloudnet1003, cloudvirt1017/1020/1022 for PDU operations in the rack [[phab:T227542|T227542]]

=== 2019-11-04 ===
* 21:55 andrewbogott: deleting a ton of wikitech hiera pages that were either no-ops or refer to nonexistent VMs or prefixes

=== 2019-10-31 ===
* 11:01 arturo: icinga-downtimed cloudvirt1030 and cloudservices1003 for 1h due to PDU upgrade operations [[phab:T227543|T227543]]

=== 2019-10-30 ===
* 22:43 jeh: reboot cloud-bootstrapvz-stretch to resolve bad bootstrapvz build

=== 2019-10-29 ===
* 10:52 arturo: icinga downtime cloudvirt1001/1002/1024/1018/1012/1009/1015/1008 for 1h [[phab:T227538|T227538]]

=== 2019-10-25 ===
* 10:45 arturo: icinga downtime toolschecker for 1 to upgrade clouddb1002 mariadb (toolsdb secondary) ([[phab:T236384|T236384]] , [[phab:T236420|T236420]])

=== 2019-10-24 ===
* 12:30 arturo: starting cloudvirt1019, PDU operations ended ([[phab:T227540|T227540]])
* 11:58 arturo: icinga downtime for 2h ([[phab:T227540|T227540]]) cloudvirt1019
* 11:15 arturo: poweroff cloudvirt1019 during the PDU operations ([[phab:T227540|T227540]])
* 11:10 arturo: icinga downtime for 2h ([[phab:T227540|T227540]]) toolschecker
* 10:58 arturo: icinga downtime for 1h ([[phab:T227540|T227540]]) cloudvirt100[3-7], cloudvirt1019, cloudvirt1016, cloudvirt1021, cloudvirt1013, cloudnet1004

=== 2019-10-23 ===
* 09:23 arturo: cloudvirt1026 reboot ended OK
* 09:12 arturo: rebooting cloudvirt1026 for kernel upgrade
* 09:09 arturo: cloudvirt1025 reboot ended OK
* 09:00 arturo: rebooting cloudvirt1025 for kernel upgrade
* 08:51 arturo: icinga downtime cloudvirt1025/1026 for reboots

=== 2019-10-18 ===
* 16:01 arturo: created the `eqiad1.wikimedia.cloud` DNS zone ([[phab:T235846|T235846]])
* 14:27 andrewbogott: deleted a bunch of leaked VMS from earlier today from the admin-monitoring project.   Fullstack leaks due to an api outage, maybe?
* 10:44 arturo: double max_message_size from 40KB to 80KB in the cloud-admin mailing list. A simple email with a couple of quotes can go over the 40KB limit.

=== 2019-10-16 ===
* 21:59 jeh: resync wiki replica tool and user accounts [[phab:T235697|T235697]]
* 09:40 arturo: reboot of cloudvirt1030 went fine
* 09:28 arturo: reboot of cloudvirt1029 went fine
* 09:28 arturo: rebooting cloudvirt1030 for kernel updates
* 09:12 arturo: rebooting cloudvirt1029 for kernel updates
* 09:11 arturo: reboot of cloudvirt1028 went fine
* 09:00 arturo: rebooting cloudvirt1028 for kernel updates
* 08:56 arturo: icinga downtime cloudvirt[1028-1030].eqiad.wmnet for 1h for reboots

=== 2019-10-15 ===
* 13:30 jeh: creating indexes and views for banwiki [[phab:T234770|T234770]]

=== 2019-10-10 ===
* 18:55 bd808: Created indexes and views for nqowiki ([[phab:T230543|T230543]])
* 11:59 arturo: network switch hardware is down affecting cloudvirt1025/1026 ([[phab:T227536|T227536]]) VMs are supposed to be online but unreachable

=== 2019-10-09 ===
* 10:44 arturo: cloudvirt1013 rebooted well
* 10:32 arturo: cloudvirt1013 is rebooting
* 10:32 arturo: cloudvirt1012 rebooted just fine (very slow, 35 VMs)
* 10:21 arturo: cloudvirt1012 is rebooting
* 10:19 arturo: cloudvirt1009 rebooted just fine (very slow though)
* 10:07 arturo: cloudvirt1009 is rebooting
* 10:06 arturo: cloudvirt1008 rebooted just fine (very slow though)
* 09:58 arturo: cloudvirt1008 is rebooting
* 09:52 arturo: icinga downtime toolschecker, paws, etc for 2h, because cloudvirt reboots

=== 2019-10-07 ===
* 14:07 arturo: horizon is disabled for maintenance ([[phab:T212302|T212302]])
* 14:00 arturo: starting scheduled maintenance: upgrading eqiad1 from openstack mitaka to newton

=== 2019-10-02 ===
* 15:23 arturo: codfw1dev renaming net/subnet objects to a more modern naming scheme [[phab:T233665|T233665]]
* 12:49 arturo: codfw1dev delete all floating ip allocations in the deployment for mangling the network config for testing [[phab:T233665|T233665]]
* 12:47 arturo: codfw1dev deleting all VMs in the deployment for mangling the network config for testing [[phab:T233665|T233665]]
* 11:08 arturo: codfw1dev rebooting cloudnet2002-dev and cloudnet2003-dev for testing [[phab:T233665|T233665]]
* 10:31 arturo: codfw1dev: add cloudinstances2b-gw router to the l3 agent in cloudnet2003-dev
* 09:59 arturo: codfw1dev: cleanup leftover &quot;HA port tenant admin&quot; in neutron (ports from missing servers)
* 09:46 arturo: codfw1dev: cleanup leftover neutron agents

=== 2019-09-30 ===
* 10:21 arturo: we installed ferm in every VM by mistake. Deleting it and forcing a puppet agent run to try to go back to a clean state.
* 09:38 arturo: downtime toolschecker for 24h
* 09:33 arturo: force update ferm cloud-wide (in all VMs) for [[phab:T153468|T153468]]

=== 2019-08-18 ===
* 10:39 arturo: rebooting cloudvirt1023 for new interface names configuration
* 10:34 arturo: downtimed cloudvirt1023 for 2 days

=== 2019-08-05 ===
* 17:17 bd808: Set downtime on gridengine and kubernetes webservice checks in icinga until 2019-09-02 (flaky tests)

=== 2019-07-29 ===
* 20:14 bd808: Restarted maintain-kubeusers on tools-k8s-master-01 ([[phab:T194859|T194859]])

=== 2019-07-25 ===
* 12:32 arturo: eqiad1/glance: debian-9.9-stretch image deprecates debian-9.8-stretch ([[phab:T228983|T228983]])
* 09:59 arturo: (codfw1dev) drop missing glance images ([[phab:T228972|T228972]])
* 09:32 arturo: (codfw1dev) deleting a bunch of VMs that were running in now missing hypervisors
* 09:31 arturo: (codfw1dev) deleting a bunch of VMs in ERROR and SHUTDOWN state
* 09:27 arturo: last log entry refers to the codfw1dev deployment
* 09:27 arturo: cleanup `nova service-list` from old hypervisors (labtest*)
* 09:23 arturo: refreshed nova DB grants in clouddb2001-dev for the codfw1dev deployment
* 08:47 arturo: cleanup the cloud-announce pending emails (spam)

=== 2019-07-23 ===
* 19:43 andrewbogott: restarting rabbitmq-server on cloudcontrol1003 and 1004

=== 2019-07-22 ===
* 23:44 bd808: Restarted maintain-kubeusers on tools-k8s-master-01 ([[phab:T228529|T228529]])

=== 2019-07-11 ===
* 22:07 bd808: Ran `sudo systemctl stop designate_floating_ip_ptr_records_updater.service` on cloudcontrol1003
* 22:01 bd808: `sudo apt-get install python2.7-dbg` on cloudcontrol1003 to debug hung python process
* 21:48 bd808: Ran `sudo systemctl stop designate_floating_ip_ptr_records_updater.service` on cloudcontrol1004

=== 2019-06-25 ===
* 16:05 bstorm_: updated python3.4 to update4 wherever it was installed on Jessie VMs to prevent issues with broken update3.
* 14:56 bstorm_: Updated python 3.4 on the labs-puppetmaster server

=== 2019-06-03 ===
* 15:55 arturo: [[phab:T221769|T221769]] rebooting cloudservices1003 after bootstrapping is apparently completed

=== 2019-05-28 ===
* 21:42 bstorm_: unmounting labstore1003-scratch on all cloud clients
* 18:14 bstorm_: [[phab:T209527|T209527]] switched mounts from labstore1003 to cloudstore1008 for scratch

=== 2019-05-20 ===
* 17:25 arturo: [[phab:T223923|T223923]] dropped compat-network config from /etc/network/interfaces in eqiad1/codfw1dev neutron nodes
* 17:22 arturo: [[phab:T223923|T223923]] dropped br-compat bridges and vlan interfaces (1102 and 2102) in eqiad1/codfw1dev neutron nodes
* 17:07 arturo: [[phab:T223923|T223923]] dropped compat-network configuration from the neutron database in eqiad1
* 16:55 arturo: [[phab:T223923|T223923]] dropped compat-network configuration from the neutron database in codfw1dev

=== 2019-05-15 ===
* 17:00 andrewbogott: touching /root/firstboot_done on all VMs that cumin can reach.  This will prevent firstboot.sh from running a second time if/when any of these are rebooted.  [[phab:T223370|T223370]]

=== 2019-04-26 ===
* 15:51 arturo: andrew updated dns servers for the cloud-instances2-b-eqiad subnet in neutron: 208.80.154.143 and 208.80.154.24

=== 2019-04-25 ===
* 11:14 arturo: [[phab:T221760|T221760]] increased size of conntrack table

=== 2019-04-24 ===
* 12:54 arturo: [[phab:T220051|T220051]] puppet broken in every VM in Cloud VPS, fixing right now

=== 2019-04-22 ===
* 11:14 arturo: create by hand /var/cache/labsaliaser/labs-ip-aliases.json in cloudservices2002-dev ([[phab:T218575|T218575]])

=== 2019-04-16 ===
* 22:55 bd808: cloudcontrol2003-dev: added `exit 0` to /etc/cron.hourly/keystone to stop cron spam on partially configured cluster
* 12:08 arturo: rebooting cloudvirt200[123]-dev because deep changes in config
* 11:27 arturo: [[phab:T219626|T219626]] add DB grants for neutron and glnace to clouddb2001-dev (codfw1dev)
* 10:37 arturo: [[phab:T219626|T219626]] replace 208.80.153.75 with 208.80.153.59 in the clouddb2001-dev database (codfw1dev deployment)
* 10:30 arturo: [[phab:T219626|T219626]] replace labtestcontrol2003 with cloudcontrol2001-dev in the clouddb2001-dev database (codfw1dev deployment)

=== 2019-04-15 ===
* 13:08 arturo: [[phab:T219626|T219626]] add DB grants for keystone/nova/nova_api to clouddb2001-dev (codfw1dev)

=== 2019-04-13 ===
* 18:25 bd808: Restarted nova-compute service on cloudvirt1015 ([[phab:T220853|T220853]])

=== 2019-04-11 ===
* 12:00 arturo: [[phab:T151704|T151704]] deploying oidentd to cloudnet1xxx servers

=== 2019-04-02 ===
* 19:52 andrewbogott: installed new base Stretch image.  Updated packages, and runs apt-get dist-upgrade on first boot.

=== 2019-03-29 ===
* 14:34 andrewbogott: moving tools-static.wmflabs.org to point to tools-static-13 in eqiad1-r
* 00:00 bstorm_: [[phab:T193264|T193264]] Added osm.db.svc.eqiad.wmflabs to cloud DNS

=== 2019-03-25 ===
* 00:40 bd808: Restarted maintain-dbusers on labstore1004. Process hung up on failed LDAP connection.

=== 2019-03-21 ===
* 19:32 andrewbogott: restarting keystone on cloudcontrol1003

=== 2019-03-15 ===
* 16:00 gtirloni: increased nscd cache size ([[phab:T217280|T217280]])

=== 2019-03-14 ===
* 19:04 gtirloni: bstorm started nfsd on labstore1006 ([[phab:T218341|T218341]])
* 16:42 gtirloni: published new debian-9.8 image ([[phab:T218314|T218314]])

=== 2019-03-04 ===
* 19:37 bstorm_: umounted /mnt/nfs/dumps-labstore1006.wikimedia.org across all VPS projects for [[phab:T217473|T217473]]

=== 2019-02-26 ===
* 12:46 gtirloni: shutdown toolsbeta-sgegrid-master (cronspam)

=== 2019-02-25 ===
* 10:32 gtirloni: restarted nfsd on labstore1004

=== 2019-02-21 ===
* 09:09 gtirloni: restarted uwsgi-labspuppetbackend.service on labpuppetmaster1001
* 07:42 gtirloni: created project cloudstore
* 07:36 gtirloni: deleted wmcs-nfs project

=== 2019-02-20 ===
* 21:58 andrewbogott: silencing shinken and disabling puppet on shinken-02 for now

=== 2019-02-19 ===
* 12:00 gtirloni: added nagios@icinga2001.wikimedia.org to cloud-admin-feed@ allowed senders

=== 2019-02-18 ===
* 20:21 gtirloni: downtimed cloudvirt1020
* 20:12 gtirloni: ran `labs-ip-alias-dump.py` on cloudservices/labservices servers

=== 2019-02-15 ===
* 13:10 arturo: [[phab:T216239|T216239]] labvirt1019 has been drained
* 12:22 arturo: [[phab:T216239|T216239]] draining labvirt1009 with a command like this: `root@cloudcontrol1004:~# wmcs-cold-migrate --region eqiad --nova-db nova 2c0cf363-c7c3-42ad-94bd-{{Gerrit|e586f2492321}} labvirt1001`
* 12:02 arturo: more nova service cleanups in the database (labvirts that were reallocated to eqiad1)
* 11:34 arturo: [[phab:T216190|T216190]] cleanup from nova database `nova service-delete 35`
* 03:50 andrewbogott: updated VPS base images for Jessie and Stretch, now featuring Stretch 9.7

=== 2019-02-11 ===
* 18:13 gtirloni: cleaned old metrics data in labmon1001 [[phab:T215417|T215417]]
* 15:28 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1011
* 14:18 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1010

=== 2019-02-08 ===
* 14:56 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1009

=== 2019-02-06 ===
* 11:47 gtirloni: downtimed labmon100{1,2} [[phab:T215399|T215399]]
* 00:17 bstorm_: [[phab:T214106|T214106]] deleted bstorm-test2 project to clean up

=== 2019-02-05 ===
* 10:48 arturo: labmon1001 is now part of the 'eqiad1-r' region

=== 2019-02-01 ===
* 09:54 arturo: moving canary1015-01 VM instance from cloudvirt1024 back to cloudvirt1015

=== 2019-01-31 ===
* 12:44 arturo: [[phab:T215012|T215012]] depooling cloudvirt1015 and migrating all VMs to cloudvirt1024

=== 2019-01-25 ===
* 20:11 gtirloni: deleted project yandex-proxy [[phab:T212306|T212306]]
* 20:11 gtirloni: deleted project [[phab:T212306|T212306]]

=== 2019-01-24 ===
* 11:50 arturo: [[phab:T213925|T213925]] modify subnet cloud-instances-transport1-b-eqiad1 to avoid floating IP allocations from here
* 11:07 arturo: [[phab:T214299|T214299]] failover cloudnet1003 to cloudnet1004
* 10:03 arturo: [[phab:T214299|T214299]] reimage cloudnet1004 to debian stretch
* 09:51 arturo: [[phab:T214299|T214299]] failover cloudnet1004 to cloudnet1003

=== 2019-01-22 ===
* 19:19 arturo: [[phab:T214299|T214299]] stretch cloudnet1003 is apparently all set
* 18:40 arturo: [[phab:T214299|T214299]] manually delete from neutron agents from cloudnet1003 (must be added again after reimage, with new uuids)
* 18:37 arturo: [[phab:T214299|T214299]] reimaging cloudnet1003 as debian stretch
* 17:35 jbond42: starting roll out of apt package updates to
* 14:41 gtirloni: [[phab:T214369|T214369]] deployed new jessie and stretch VM images

=== 2019-01-21 ===
* 18:29 gtirloni: installed libguestfs-tools on cloudvirt1021

=== 2019-01-16 ===
* 14:21 andrewbogott: stopping old VPS proxies in eqiad — [[phab:T213540|T213540]]

=== 2019-01-15 ===
* 14:20 andrewbogott: changing tools.wmflabs.org to point to tools-proxy-03 in eqiad1

=== 2019-01-13 ===
* 20:00 andrewbogott: VPS proxies are now running in eqiad1 on proxy-01.  Old VMs will wait a bit for deletion.  [[phab:T213540|T213540]]
* 19:12 andrewbogott: moving the VPS proxy API backend to proxy-01.project-proxy.eqiad.wmflabs, as per [[phab:T213540|T213540]]
* 17:11 andrewbogott: moving all VPS dynamic proxies to proxy-eqiad1.wmflabs.org aka proxy-01.project-proxy.eqiad.wmflabs, as per [[phab:T213540|T213540]]

=== 2019-01-09 ===
* 22:21 bd808: neutron quota-update --tenant-id tools --port 256

=== 2019-01-08 ===
* 18:59 bd808: Definately did NOT delete uid=novaadmin,ou=people,dc=wikimedia,dc=org
* 18:59 bd808: Deleted LDAP user uid=neutron,ou=people,dc=wikimedia,dc=org
* 18:58 bd808: Deleted LDAP user uid=novaadmin,ou=people,dc=wikimedia,dc=org

=== 2019-01-06 ===
* 22:03 bd808: Set floatingip quota of 60 for tools project in eqiad1-r region ([[phab:T212360|T212360]])

=== 2018-12-20 ===
* 17:10 arturo: [[phab:T207663|T207663]] renumbered transport network in eqiad1

=== 2018-12-05 ===
* 17:59 arturo: [[phab:T207663|T207663]] changed labtestn transport network addressing from private to public

=== 2018-12-03 ===
* 13:25 arturo: [[phab:T202886|T202886]] create again PTR records after dnsleak.py fix

=== 2018-11-30 ===
* 14:08 arturo: running dns leaks cleanup `root@cloudcontrol1003:~# /root/novastats/dnsleaks.py --delete`

=== 2018-11-28 ===
* 17:33 gtirloni: deleted contintcloud project ([[phab:T209644|T209644]])

=== 2018-11-27 ===
* 13:32 gtirloni: enabled DRBD stats collection on labstore100[4-5] [[phab:T208446|T208446]]

=== 2018-11-22 ===
* 07:12 gtirloni: deployed new debian-9.6-stretch image

=== 2018-11-21 ===
* 10:48 arturo: re-created compat-net as not shared in labtestn to test stuff related to [[phab:T209954|T209954]]

=== 2018-11-16 ===
* 12:43 gtirloni: armed keyholder on labpuppetmaster1001/1002 after reboots
* 12:08 gtirloni: rebooted labpuppetmaster1001 ([[phab:T207377|T207377]])
* 11:57 gtirloni: rebooted labpuppetmaster1002 ([[phab:T207377|T207377]])

=== 2018-11-14 ===
* 17:19 gtirloni: added cloudvirt1016 to scheduler pool ([[phab:T209426|T209426]])
* 15:41 gtirloni: reimaging labvirt1016 as cloudvirt1016
* 15:14 gtirloni: reset-failed systemd unit nova-scheduler on cloudcontrol1004
* 13:52 gtirloni: rebooted labservices1002 after package upgrades ([[phab:T207377|T207377]])
* 13:23 gtirloni: rebooted labstore2004 after package upgrades ([[phab:T207377|T207377]])
* 13:20 gtirloni: rebooted labstore2003 after package upgrades ([[phab:T207377|T207377]])
* 13:20 gtirloni: rebooted labstore2001/labstore2003 after package upgrades ([[phab:T207377|T207377]])
* 12:08 gtirloni: rebooted labnet1002 after package upgrades
* 12:01 gtirloni: rebooted labmon1002 after package upgrades
* 11:41 gtirloni: rebooted labcontrol1002 after package upgrades
* 11:15 gtirloni: rebooted cloudcontrol1004 after package upgrades

=== 2018-11-09 ===
* 18:17 gtirloni: restarted neutron-linuxbridge-agent on cloudvirt1018/1023

=== 2018-11-08 ===
* 11:00 gtirloni: Added novaproxy-02 to $CACHES
* 10:50 gtirloni: Added cloudvirt1017 to eqiad1 region

=== 2018-11-07 ===
* 13:49 arturo: [[phab:T208733|T208733]] moving labvirt1017 from main deployment to eqiad1 and renaming it to cloudvirt1017

=== 2018-10-22 ===
* 16:24 arturo: [[phab:T206261|T206261]] another update to dmz_cidr in eqiad1
* 10:26 arturo: change again in dmz_cidr in eqiad1: VMs will connect between them without NAT even when using floating IPs ([[phab:T206261|T206261]])

=== 2018-10-19 ===
* 12:02 arturo: revert change in dmz_cidr in eqiad1 for now ([[phab:T206261|T206261]])
* 11:16 arturo: change in dmz_cidr in eqiad1: VMs will connect between them without NAT even when using floating IPs ([[phab:T206261|T206261]])
* 10:14 arturo: we have new virt servers in the eqiad1 deployment since past week and this week: cloudvirt1018, cloudvirt1023, cloudvirt1024

=== 2018-09-26 ===
* 10:40 arturo: [[phab:T205524|T205524]] all sorts of restarts in all neutron daemons
* 10:20 arturo: [[phab:T205524|T205524]] stop/start all neutron agents in cloudnet1003.eqiad.wmnet
* 10:13 arturo: [[phab:T205524|T205524]] restart all agents in cloudnet1004.eqiad.wmnet
* 10:10 arturo: restart neutron-server in cloudcontrol1003, investigating [[phab:T205524|T205524]]

=== 2018-09-24 ===
* 10:57 arturo: try to increase floating ip allocation pool in eqiad1. Of 185.15.56.0/25 we are using only 185.15.56.10-185.15.56.31, I don't know why. Let's use 185.15.56.2-185.15.56.126

=== 2018-09-21 ===
* 17:18 bd808: Running `sudo maintain-meta_p --all-databases --purge` across labsdb10(09{{!}}10{{!}}11) for [[phab:T201890|T201890]]

=== 2018-09-17 ===
* 22:08 bd808: Granted gtirloni project roles of admin, projectadmin, and user

=== 2018-09-12 ===
* 11:20 arturo: [[phab:T202636|T202636]] distributing default routes using classless-static-route for all VMs in main/labtest (dnsmasq/nova-network)

=== 2018-09-11 ===
* 16:52 arturo: again, restarted nova-network after killing all dnsmasq procs in labnet1001 for [[phab:T202636|T202636]]
* 16:08 arturo: restarted nova-network after killing all dnsmasq procs in labnet1001 for [[phab:T202636|T202636]]
* 10:53 arturo: [[phab:T202636|T202636]] creating all the compat-network configuration in neutron
* 10:36 arturo: [[phab:T202636|T202636]] creating br-compat bridge in eqiad1 for the compat network
* 10:33 arturo: [[phab:T202636|T202636]] manually reserve 10.68.23.253 (in nova-network)

=== 2018-09-10 ===
* 22:46 andrewbogott: deleting all VMs on labvirt1019 and 1020 as prep for [[phab:T204003|T204003]]

=== 2018-08-30 ===
* 15:46 andrewbogott: restarting rabbitmq-server on cloudcontrol1003
* 13:07 arturo: [[phab:T202636|T202636]] internal network routing now exists in labtest/labtestn for VM to communicate with each other

=== 2018-08-28 ===
* 11:04 arturo: [[phab:T202549|T202549]] eqiad1 databases are all now running in m5-master. Mysql has been cleaned from cloudcontrol100[3,4]

=== 2018-08-23 ===
* 16:17 arturo: [[phab:T188589|T188589]] bstorm_ merged patch to reduce nova DB connection usage
* 13:15 arturo: [[phab:T202115|T202115]] `root@cloudcontrol1003:~# neutron subnet-update --allocation-pool start=10.64.22.4,end=10.64.22.4 e4fb2771-a361-4add-ac4e-280cc300c59f`
* 13:10 arturo: [[phab:T202115|T202115]] (was `{&quot;start&quot;: &quot;10.64.22.2&quot;, &quot;end&quot;: &quot;10.64.22.254&quot;}` )
* 13:08 arturo: [[phab:T202115|T202115]] `root@cloudcontrol1003:~# neutron subnet-update --allocation-pool start=10.64.22.254,end=10.64.22.254 e4fb2771-a361-4add-ac4e-280cc300c59f`

=== 2018-08-22 ===
* 15:28 arturo: cleanup local glance,keystone databases in cloudcontrol1003.wikimedia.org (already in m5-master)
* 15:27 arturo: cleanup local keystone database in cloudcontrol1003.wikimedia.org (already in m5-master)

=== 2018-08-21 ===
* 15:39 andrewbogott: initial test message
* 10:31 arturo: eqiad1 remove leftover port for HA on labnet1004
* 10:15 arturo: test

=== 2018-05-07 ===
* 18:07 bstorm_: stopped the toolhistory job because it is totally broken and fills /tmp.

=== 2018-02-09 ===
* 00:55 bd808: Added Arturo Borrero Gonzalez and Bstorm as project members
* 00:54 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])

{{SAL|Project Name=admin}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>kvsg8yzbyuu6uopkchnuqgiobnmjo4w</sha1>
    </revision>
    <revision>
      <id>2309616</id>
      <parentid>2309615</parentid>
      <timestamp>2025-06-07T18:55:17Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services</comment>
      <origin>2309616</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1268626" sha1="6li6xrsabi60h8xwjcdcika14f87f6u" xml:space="preserve">=== 2025-06-07 ===
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova

=== 2025-06-06 ===
* 20:59 andrewbogott: restarting all designate services on all cloudcontrols in eqiad1

=== 2025-06-05 ===
* 20:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,octavia
* 15:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:52 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2025-06-03 ===
* 22:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 20:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:44 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:43 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 17:50 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:36 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-06-02 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 21:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237
* 21:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242
* 17:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 15:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 00:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T394333|T394333]])

=== 2025-06-01 ===
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T394333|T394333]])

=== 2025-05-31 ===
* 23:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services ([[phab:T395742|T395742]])
* 23:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services ([[phab:T395742|T395742]])
* 23:38 andrewbogott: failing over from cloudnet1005 to 1006 in hopes of unsticking [[phab:T395742|T395742]]
* 23:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,neutron ([[phab:T395742|T395742]])
* 23:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,neutron ([[phab:T395742|T395742]])

=== 2025-05-30 ===
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-28 ===
* 22:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1076.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1076.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1075.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1075.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1074.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1074.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1073.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1073.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1072.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1072.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1071.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1071.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1070.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1070.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1069.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1069.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1068.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1068.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudrabbot1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbot1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,designate
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate

=== 2025-05-26 ===
* 13:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet2006-dev.codfw.wmnet
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet2006-dev.codfw.wmnet
* 13:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet2005-dev.codfw.wmnet
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet2005-dev.codfw.wmnet
* 12:54 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=99)
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.roll_reboot_cloudnets
* 12:49 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=99)
* 12:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.roll_reboot_cloudnets
* 12:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::control&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::control&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 07:52 taavi: add new /dev/sdb back to software raid after it was replaced in cloudcephmon1004

=== 2025-05-23 ===
* 09:00 dhinus: failover dumps_dist_active_vps to clouddumps1001 ([[phab:T383723|T383723]])

=== 2025-05-20 ===
* 19:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 15:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 14:57 andrewbogott: resetting eqiad1 rabbitmq in an attempt to resolve [[phab:T394790|T394790]]
* 03:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 03:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 02:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 00:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 00:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])

=== 2025-05-19 ===
* 22:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 22:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 22:43 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for service: project,nova
* 22:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova
* 22:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:26 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for service: project,nova
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 18:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T394727|T394727]])
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1075.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1075.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1072.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1072.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1076.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1076.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1073.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1073.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-15 ===
* 15:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 14:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 13:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-14 ===
* 17:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 17:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/231
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/231
* 12:59 taavi: powercycle unresponsive cloudnet2006-dev

=== 2025-05-12 ===
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 08:17 taavi: powercycle clouservices2005-dev.codfw.wmnet
* 03:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 03:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 02:36 andrewbogott: rebooting cloudnet2005-dev from mgmt  -- ssh is failing and the console shows a user prompt but not a password prompt.

=== 2025-05-11 ===
* 13:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova

=== 2025-05-07 ===
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 19:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:19 andrewbogott: upgrading codfw1dev to version 'epoxy' [[phab:T390914|T390914]]
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T390914|T390914]])
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T390914|T390914]])
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum,heat
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum,heat
* 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum,heat
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum,heat
* 16:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum
* 16:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,nova
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 13:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,neutron
* 13:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,neutron
* 13:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,neutron
* 13:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,neutron
* 12:48 taavi: updating all security group rules referencing old 172.16.0.0/21 subnet to reference new ip space instead ([[phab:T379175|T379175]])
* 10:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-05-06 ===
* 03:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)

=== 2025-05-05 ===
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 03:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 00:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 00:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 00:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-05-04 ===
* 23:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:53 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:41 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2025-05-03 ===
* 02:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])

=== 2025-05-02 ===
* 23:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 21:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 16:20 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=97) ([[phab:T393196|T393196]])
* 16:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 16:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 16:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrewbogott: sudo cumin --force O&lt;nowiki&gt;{&lt;/nowiki&gt;*&lt;nowiki&gt;}&lt;/nowiki&gt; &quot;dpkg --list {{!}} grep puppetserver &amp;&amp; systemctl restart puppetserver.service&quot; # work around a package update that is causing some puppetservers to error out

=== 2025-04-29 ===
* 13:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225
* 12:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225
* 11:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:55 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223
* 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223
* 07:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 07:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221

=== 2025-04-28 ===
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:21 taavi: migrating opentofu managed default security group rules [[phab:T392799|T392799]]
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-26 ===
* 07:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0) ([[phab:T390134|T390134]])
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance ([[phab:T390134|T390134]])
* 07:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T390134|T390134]])
* 07:47 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T390134|T390134]])

=== 2025-04-25 ===
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218
* 12:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 04:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T390134|T390134]])

=== 2025-04-24 ===
* 17:25 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T390134|T390134]])
* 13:29 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for service: project,designate
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate

=== 2025-04-23 ===
* 21:45 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 21:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:45 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:21 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 19:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:32 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:29 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:29 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:17 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:27 arturo: enabling IPv6 dualstack on neutron virtual router ([[phab:T380174|T380174]])
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:14 arturo: enable IPv6 on cloudgw ([[phab:T380174|T380174]]) -- includes server reboot

=== 2025-04-22 ===
* 04:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)

=== 2025-04-21 ===
* 23:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 23:34 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 23:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 23:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 22:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:09 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 22:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 16:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 01:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 01:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-16 ===
* 16:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 arturo: merging network change in neutron https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/198
* 10:10 dcaro: upgrade spicerack on cloudcumin2001 to 10.1.0
* 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 10:07 dcaro: upgrade spicerack on cloudcumin1001 to 10.1.0

=== 2025-04-15 ===
* 15:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:38 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:31 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:30 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-14 ===
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 17:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:50 andrewbogott: granting 'tofuadmin' user inherited 'member' role in all projects, this should fix some policy mishaps
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-13 ===
* 22:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 22:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-11 ===
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-10 ===
* 12:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:05 arturo: [codfw1dev] root@cloudcontrol2004-dev:~# wmcs-makedomain --project testlabs --domain testlabs.codfw1dev.wmcloud.org --orig-project cloudinfra-codfw1dev ([[phab:T391325|T391325]])

=== 2025-04-09 ===
* 23:23 bd808: Rebooting tools-sgebastion-10 (login-buster.toolforge.org) for high load/unresponsive NFS mounts ([[phab:T391538|T391538]])
* 10:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 08:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 04:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 03:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:30 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 03:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services

=== 2025-04-08 ===
* 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet'
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet'
* 22:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1011.eqiad.wmnet'
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet'
* 11:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2025-04-07 ===
* 23:43 bd808: `sudo service maintain-dbusers restart` on cloudcontrol1007 after reports of missing replica.my.cnf and finding the journal for the service empty.
* 15:30 arturo: [codfw1dev] testlabs create a bunch of VMs by hand, like `networktests-vlan-legacy-floating` [[phab:T380728|T380728]]
* 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:23 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-06 ===
* 02:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 02:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-04 ===
* 15:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 05:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 05:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:10 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for all services
* 05:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:04 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for all services
* 05:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-03 ===
* 22:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 20:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 20:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 20:11 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:29 taavi: run wmcs-wikireplica-dns to create transitional x3 CNAMEs [[phab:T390954|T390954]]

=== 2025-04-02 ===
* 20:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup2004.codfw.wmnet' ([[phab:T381499|T381499]])
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.unset_maintenance (exit_code=0) ({{Gerrit|1133432}})
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.unset_maintenance ({{Gerrit|1133432}})
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:22 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:26 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T381499|T381499]])
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T381499|T381499]])
* 09:53 dhinus: systemctl restart maintain-dbusers.service (attempting to fix a weird issue)

=== 2025-04-01 ===
* 21:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate
* 21:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 21:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 21:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 21:00 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-31 ===
* 15:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 15:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 15:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 15:53 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2025-03-27 ===
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 14:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 12:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T390134|T390134]])

=== 2025-03-26 ===
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-25 ===
* 14:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-24 ===
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-22 ===
* 03:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,designate
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate

=== 2025-03-19 ===
* 14:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,designate
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 12:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,task_id,no_dologmsg,cluster_name,all_services,nova,glance,keystone,cinder,neutron,trove,magnum,heat,swift,designate,filter_nodes
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,task_id,no_dologmsg,cluster_name,all_services,nova,glance,keystone,cinder,neutron,trove,magnum,heat,swift,designate,filter_nodes
* 06:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 06:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 06:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 06:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet'
* 06:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 06:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet'
* 06:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet'
* 06:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 06:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet'
* 06:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 06:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 05:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet'
* 05:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 05:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 05:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 05:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 04:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 04:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 04:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet'
* 04:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 04:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 04:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet'
* 04:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet'
* 04:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet'
* 04:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet'
* 04:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet'
* 04:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 03:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T381499|T381499]])
* 02:12 andrewbogott: upgrading codfw1dev to openstack 'dalmation'  [[phab:T381499|T381499]]
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T381499|T381499]])

=== 2025-03-14 ===
* 13:36 volans: installed cumin v5.1.1 on cloudcumin* hosts

=== 2025-03-05 ===
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 00:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 00:43 andrewbogott: restarting all openstack services in hopes of gettting dns unstuck
* 00:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services

=== 2025-03-04 ===
* 08:52 arturo: [[phab:T387828|T387828]] depooled galera on cloudcontrol1005

=== 2025-03-01 ===
* 19:35 andrewbogott: installed new bookworm base images in eqiad1
* 19:35 andrewbogott: installed new bookworm and bullseye base images in codfw1dev

=== 2025-02-28 ===
* 15:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-27 ===
* 15:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-24 ===
* 00:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2025-02-23 ===
* 21:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-21 ===
* 12:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-20 ===
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T386083|T386083]])
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T386083|T386083]])

=== 2025-02-19 ===
* 13:26 arturo: manual failover of cloudgw1004 to cloudgw1003 [[phab:T382356|T382356]]
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-18 ===
* 12:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-02-13 ===
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-11 ===
* 11:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T386083|T386083]])
* 11:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T386083|T386083]])
* 11:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047' ([[phab:T386083|T386083]])
* 11:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047' ([[phab:T386083|T386083]])

=== 2025-02-07 ===
* 13:52 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet1005.eqiad.wmnet ([[phab:T384946|T384946]])
* 13:48 root@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet1005.eqiad.wmnet ([[phab:T384946|T384946]])
* 02:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 01:48 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2025-02-06 ===
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:45 andrewbogott: cold-migrating all remaining VMs in [[phab:T385264|T385264]] except for 'integration' and 'tools' VMs

=== 2025-02-05 ===
* 19:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 19:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2025-02-04 ===
* 12:48 arturo: replacing cloudgw1002 with cloudgw1004 - [[phab:T382356|T382356]]
* 09:22 arturo: fleet-wide restart of puppetservers [[phab:T385553|T385553]]

=== 2025-02-03 ===
* 13:42 andrewbogott: rebooting proxy-04.project-proxy for the ceph OSD mishap

=== 2025-02-02 ===
* 17:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 17:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2025-01-31 ===
* 01:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-01-30 ===
* 21:39 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 21:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:33 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 12:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1031.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1031.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 03:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 02:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 02:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 00:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 00:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)

=== 2025-01-29 ===
* 23:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 23:43 andrewbogott: resetting rabbitmq in eqiad1 in hopes that it will resolve mysterious openstack misbehaviors
* 20:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2009-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2009-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1007.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1007.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1005.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1005.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-28 ===
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:51 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 13:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 13:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 13:02 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T348643|T348643]])
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 00:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 00:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T348643|T348643]])
* 00:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2025-01-27 ===
* 21:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:52 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 14:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-24 ===
* 13:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node

=== 2025-01-23 ===
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 20:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 15:49 dhinus: cumin 'P:base::cloud_production' 'rm /var/lib/prometheus/node.d/kernel-panic.prom' [[phab:T382961|T382961]]
* 15:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-22 ===
* 14:52 dhinus: cloudcumin[12]001 upgrade spicerack from 8.15.2 to 9.1.0

=== 2025-01-21 ===
* 13:42 andrewbogott: migrating/rebooting VMs as per earlier email,  [[phab:T383583|T383583]]

=== 2025-01-20 ===
* 14:23 dhinus: cumin upgraded form 4.2.0 to 5.0.0 on cloudcumin[12]001. patch [[phab:T346453|T346453]] reapplied.

=== 2025-01-14 ===
* 21:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T383583|T383583]])
* 21:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T383583|T383583]])
* 20:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T383583|T383583]])
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T383583|T383583]])
* 20:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T383583|T383583]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T383583|T383583]])
* 18:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.roll_restart_osd_daemons (exit_code=0)

=== 2025-01-13 ===
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.roll_restart_osd_daemons

=== 2025-01-10 ===
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-01-09 ===
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2025-01-08 ===
* 14:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 13:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2025-01-03 ===
* 21:01 bd808: `sudo service maintain-dbusers restart` on cloudcontrol1005. Report of missing replica.my.cnf and journalctl output empty due to log rotation. ([[phab:T382962|T382962]])

=== 2024-12-18 ===
* 15:33 arturo: cloudgw failover for [[phab:T382220|T382220]]

=== 2024-12-04 ===
* 17:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 17:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 15:09 andrewbogott: rebooted cloudinfra-cloudvps-puppetserver-1, it's so busy that it's unresponsive
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])

=== 2024-11-27 ===
* 13:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet'
* 13:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'

=== 2024-11-26 ===
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 15:16 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:15 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:15 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:15 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:14 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/145
* 15:13 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/145
* 15:12 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:12 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:31 dcaro: added cloudcephmon1004 to the ceph mon pool
* 12:41 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 05:40 andrewbogott: rebooting tools-sgebastion-10.tools.eqiad1.wikimedia.cloud to get NFS things remounted
* 04:56 andrewbogott: 'systemctl restart nfs-server' on tools-nfs-2.tools.eqiad1.wikimedia.cloud

=== 2024-11-25 ===
* 14:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:40 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 13:55 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:28 arturo: create IPv6 networks in eqiad1 ([[phab:T380174|T380174]]), then reverted because network outage
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-23 ===
* 14:58 taavi: removed broken records for re-created VMs in .eqiad.wmflabs zone

=== 2024-11-22 ===
* 14:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:35 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:48 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-20 ===
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2024-11-19 ===
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:43 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:43 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:37 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:35 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 11:39 arturo: [codfw1dev] performing rabbit full reset [[phab:T380208|T380208]]
* 10:26 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:24 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 10:24 arturo: [codfw1dev] restart rabbitmq and nova/neutron services for [[phab:T380208|T380208]]

=== 2024-11-16 ===
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 07:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-15 ===
* 19:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:10 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:30 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:29 arturo: [codfw1dev] restart rabbitmq and designate
* 16:29 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-14 ===
* 09:52 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:48 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-13 ===
* 13:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/120
* 13:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/120

=== 2024-11-11 ===
* 14:49 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-11-08 ===
* 16:47 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:45 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 16:42 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:42 arturo: [codfw1dev] restart all nova services and rabbitmq out of despair
* 16:42 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:45 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:44 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:33 aborrero@cloudcumin2001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 13:32 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:28 arturo: [codfw1dev] restart rabbitmq, openstack services logs show connection errors

=== 2024-11-07 ===
* 17:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 17:42 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-06 ===
* 13:09 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 11:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/116
* 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/116
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-05 ===
* 10:24 arturo: [codfw1dev] disable puppet and make changes for testing [[phab:T378192|T378192]]

=== 2024-11-04 ===
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:09 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:08 aborrero@cloudcumin2001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:08 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 14:08 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:08 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:52 arturo: [codfw1dev] live-hack cloudlb2001-dev and cloudcontrol2004-dev for [[phab:T378192|T378192]]
* 13:52 arturo: [codfw1dev] restart rabbitmq
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-29 ===
* 15:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-28 ===
* 17:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:00 arturo: [codfw1dev] restarting rabbitmq, misbehaving
* 17:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 dhinus: apt full-upgrade and reboot for cloudcumin*
* 16:21 dhinus: upgrade spicerack from 8.8.0 to 8.15.1 on cloudcumin*

=== 2024-10-24 ===
* 15:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-22 ===
* 15:02 taavi: recover access to User:Labslogbot [[phab:T376220|T376220]]

=== 2024-10-21 ===
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:06 arturo: [codfw1dev] restart rabbitmq, tofu shows error talking to the designate API

=== 2024-10-16 ===
* 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-15 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:38 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:33 arturo: cloudgw maintenance, firewall change for [[phab:T374714|T374714]]
* 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-11 ===
* 15:31 arturo: cloudgw maintenance firewall change  [[phab:T374716|T374716]]
* 09:51 arturo: cloudgw network maintenance related to [[phab:T376879|T376879]]
* 08:33 arturo: [codfw1dev] reboot cloudgw2002-dev/2003-dev because network connectivity issues

=== 2024-10-10 ===
* 12:04 arturo: manual network failover in cloudgw because maintenance related to [[phab:T376879|T376879]]
* 10:40 dhinus: cumin 'cloudrabbit*' 'systemctl restart rabbitmq-server' [[phab:T376802|T376802]]
* 09:19 arturo: [codfw1dev] enable IPv6 on cloudgw ([[phab:T374716|T374716]])

=== 2024-10-09 ===
* 14:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:20 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:19 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:19 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:16 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:16 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:58 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:58 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:55 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:55 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:54 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:53 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:53 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95
* 10:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2024-10-08 ===
* 11:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-07 ===
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-04 ===
* 15:12 arturo: cloudservice1005/1006: enable puppet and restore /etc/powerdns/recursor.conf to non-debug mode ([[phab:T374830|T374830]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:32 arturo: cloudservice1005/1006: disable puppet and set `quiet=no` in /etc/powerdns/recursor.conf ([[phab:T374830|T374830]])

=== 2024-10-03 ===
* 14:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:54 arturo: [codfw1dev] delete default security group rule list, now tracking them via tofu-infra
* 14:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:51 arturo: delete default security group rule list, now tracking them via tofu-infra
* 14:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-02 ===
* 15:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:21 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:16 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:15 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:05 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 09:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:10 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])

=== 2024-10-01 ===
* 15:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-30 ===
* 20:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T372814|T372814]])
* 16:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 16:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 13:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 13:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T372814|T372814]])
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 13:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 11:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 11:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 11:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 11:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 10:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 10:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=99)
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=99)
* 09:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 09:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-27 ===
* 15:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:56 arturo: [codfw1dev] restart rabbitmq again
* 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:15 arturo: [codfw1dev] restart rabbitmq
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:04 arturo: [codfw1dev] enable IPv6 on the neutron virtual router [[phab:T375847|T375847]]

=== 2024-09-26 ===
* 14:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 14:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T372814|T372814]])
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 10:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T372814|T372814]])
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:47 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:45 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-25 ===
* 14:11 arturo: [codfw1dev] start proxy-02 vm on proxy-codfw1dev project, it was in shutoff mode for unknown reasons
* 10:33 arturo: [codfw1dev] cleanup unused security groups ([[phab:T375604|T375604]])
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:48 arturo: [codfw1dev] restart rabbitmq on all cloudcontrol servers
* 09:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:35 arturo: [codfw1dev] deletre a bunch of tests and seemingly unused projects

=== 2024-09-24 ===
* 19:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 16:00 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 14:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T348643|T348643]])
* 14:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 14:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:56 arturo: [codfw1dev] restart rabbitmq-server on all 3 nodes
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack
* 09:10 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_rack (exit_code=97)
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack

=== 2024-09-23 ===
* 19:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_rack (exit_code=99)
* 15:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:42 arturo: put cloudvirt1048 in the network-ovs aggregate [[phab:T364457|T364457]]
* 14:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:37 arturo: [codfw1dev] restart rabbitmq
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:23 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-09-21 ===
* 10:17 dhinus: nova host-evacuate cloudvirt1063 ([[phab:T375223|T375223]])
* 09:59 dhinus: openstack aggregate remove host ceph cloudvirt1063 ([[phab:T375223|T375223]])
* 09:59 dhinus: openstack aggregate add host maintenance cloudvirt1063 ([[phab:T375223|T375223]])
* 09:42 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1063.eqiad.wmnet'
* 09:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1063.eqiad.wmnet'

=== 2024-09-20 ===
* 11:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 10:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 08:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T373740|T373740]])
* 08:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T373740|T373740]])

=== 2024-09-19 ===
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T373740|T373740]])
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T373740|T373740]])
* 15:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) ([[phab:T373740|T373740]])
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance ([[phab:T373740|T373740]])
* 10:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51
* 10:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51
* 09:56 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T374043|T374043]])
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 09:51 arturo: [codfw1dev] play with neutron default security group rules (delete, create them, etc) [[phab:T375111|T375111]]
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50

=== 2024-09-18 ===
* 15:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:21 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 15:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/47
* 11:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/47
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:40 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:39 dcaro: restarted rabbitmq-server on all cloudrabbits
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-09-17 ===
* 21:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T374043|T374043]])
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 16:11 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T374043|T374043]])
* 16:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 15:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46

=== 2024-09-16 ===
* 14:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 11:28 arturo: [codfw1dev] created VM bastion-codfw1dev-04 to replace current bastion -03 ([[phab:T374828|T374828]])

=== 2024-09-12 ===
* 10:51 arturo: merging change to keystone wmf hooks https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071230 ([[phab:T374020|T374020]])

=== 2024-09-11 ===
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 16:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 15:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 15:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/41
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/41
* 11:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 07:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T374467|T374467]])
* 07:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T374467|T374467]])

=== 2024-09-10 ===
* 12:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 11:37 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:42 dcaro: hard-rebooting cloudvirt2004-dev (codfw1dev) having io/hardware issues
* 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 08:45 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-09 ===
* 21:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T373986|T373986]])
* 16:36 dcaro: cleaned up dns leaks
* 15:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/39
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/39
* 15:05 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 14:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:06 arturo: merged change to cloudgw NAT setting https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071189
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:42 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:38 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-06 ===
* 23:18 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T373986|T373986]])
* 18:17 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 13:46 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 07:27 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-05 ===
* 21:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 17:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T374043|T374043]])
* 14:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:19 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 14:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 12:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/36
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/36
* 10:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/35
* 10:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/35
* 09:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/34
* 09:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/34
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/33
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/33
* 09:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:22 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/31
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/31
* 09:16 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 08:46 arturo: [codfw1dev] restart rabbitmq @ codfw1dev [[phab:T374002|T374002]]

=== 2024-09-04 ===
* 21:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 19:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:35 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 12:18 arturo: [codfw1dev] restart rabbitmq-server.service on all 3 cloudcontrols, all nova-compute agents are down complaining about rabbitmq being unreachable

=== 2024-09-02 ===
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 11:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28

=== 2024-08-31 ===
* 13:55 andrewbogott: moving tools-redis-7 off of cloudvirt1048 just in case  [[phab:T373740|T373740]]
* 13:39 andrewbogott: rebooting cloudvirt1048 from mgmt, it seems to have crashed

=== 2024-08-30 ===
* 12:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-29 ===
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-08-25 ===
* 22:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-23 ===
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T369044|T369044]])
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T369044|T369044]])
* 00:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-22 ===
* 23:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=97) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.unset_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.unset_maintenance ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=97) on host 'cloudvirt1039' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1037' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1035' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035' ([[phab:T369044|T369044]])
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1033' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1052' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052' ([[phab:T369044|T369044]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])

=== 2024-08-20 ===
* 03:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)

=== 2024-08-19 ===
* 23:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 23:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 23:16 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-08-17 ===
* 03:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-16 ===
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:20 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T363344|T363344]])
* 05:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:57 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:50 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 01:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:12 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-15 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:27 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 08:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 04:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 02:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-14 ===
* 23:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:05 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-12 ===
* 15:44 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 08:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=0)
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:37 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97) ([[phab:T371878|T371878]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) ([[phab:T371878|T371878]])
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) ([[phab:T371878|T371878]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:17 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97)
* 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_rack

=== 2024-08-09 ===
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 18:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 13:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 13:38 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 13:35 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 13:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 11:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 05:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 05:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 00:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 00:47 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])

=== 2024-08-08 ===
* 23:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet'
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet'
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 18:39 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet'
* 18:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet'
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet'
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet'
* 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet'
* 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet'
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet'
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 16:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet'
* 16:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet'
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudbackup1002-dev.codfw.wmnet'
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.codfw.wmnet'
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet'
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet'
* 16:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2006-dev.codfw.wmnet'
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2006-dev.codfw.wmnet'
* 16:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet'
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 15:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 15:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.wikimedia.org'
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.wikimedia.org'
* 15:51 andrewbogott: upgrading codfw1dev to openstack version caracal https://phabricator.wikimedia.org/T369044
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 15:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 15:49 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T369044|T369044]])
* 15:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 14:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 14:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.wait_for_rebalance
* 13:00 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 12:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 12:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 12:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 11:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T371878|T371878]])
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T371878|T371878]])
* 09:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 09:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 07:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 07:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 06:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 05:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 05:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)

=== 2024-08-07 ===
* 20:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 20:11 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=97)
* 20:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 20:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 20:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 18:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 17:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:56 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=97)
* 16:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node
* 16:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node
* 16:40 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:24 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T371878|T371878]])
* 08:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T371878|T371878]])
* 06:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 03:08 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])

=== 2024-08-06 ===
* 21:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 19:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 18:36 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet'
* 18:21 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1042.eqiad.wmnet'
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet'
* 18:17 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet'
* 18:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet'
* 17:59 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet'
* 17:58 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet'
* 17:45 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet'
* 17:43 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet'
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet'
* 17:32 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet'
* 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1045.eqiad.wmnet'
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet'
* 17:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet'
* 17:21 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet'
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet'
* 17:14 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet'
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet'
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 17:01 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet'
* 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 17:00 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:48 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:47 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 16:47 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:37 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 16:37 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:25 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance

=== 2024-08-01 ===
* 13:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-07-31 ===
* 23:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 23:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 16:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27

=== 2024-07-30 ===
* 11:29 arturo: installing nova security updates ([[phab:T371240|T371240]])

=== 2024-07-29 ===
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:28 arturo: [codfw1dev] restarting rabbitmq-server on all cloudcontrols, nova-compute cannot contact it
* 11:00 arturo: [codfw1dev] installing nova security updates ([[phab:T371240|T371240]])
* 10:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27

=== 2024-07-25 ===
* 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:46 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:45 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:09 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:55 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 11:40 arturo: manually restart maintain-dbusers in cloudcontrol1005 to see if that makes any difference
* 11:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 07:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 07:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23

=== 2024-07-24 ===
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 16:01 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:46 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:43 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:37 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 11:59 arturo: restarted maintain-dbusers.service in cloudcontrol1005, it was stuck doing nothing
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:43 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:42 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24

=== 2024-07-23 ===
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 10:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 09:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 05:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 05:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 05:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 05:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 03:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 03:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 02:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 02:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)

=== 2024-07-22 ===
* 23:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 23:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 21:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 20:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 18:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 15:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:28 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 13:28 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:10 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy

=== 2024-07-21 ===
* 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 09:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 06:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 06:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 03:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 03:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 00:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-20 ===
* 21:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 18:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 18:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 18:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 06:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 06:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 03:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 01:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 00:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-19 ===
* 22:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 19:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 18:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 18:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 06:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 06:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 01:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-18 ===
* 22:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 22:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 19:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 12:31 dhinus: upgrade spicerack from 8.5 to 8.8 on cloudcumin*
* 10:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 07:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 07:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 03:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-17 ===
* 22:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 17:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 07:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 05:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 05:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 02:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 02:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-16 ===
* 23:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 23:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 17:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 11:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 11:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 09:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 09:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-07-15 ===
* 22:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:40 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 20:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 18:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy

=== 2024-07-11 ===
* 13:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 13:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 11:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 11:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 02:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'

=== 2024-07-08 ===
* 17:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T309789|T309789]])
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-07-06 ===
* 14:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-07-05 ===
* 10:33 arturo: aborrero@cloudcephmon1001:~$ sudo ceph osd unset norebalance
* 10:31 arturo: aborrero@cloudcephmon1001:~$ sudo ceph osd unset noin
* 08:56 arturo: installing nova/glance/cinder security updates [[phab:T369138|T369138]]

=== 2024-07-04 ===
* 20:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 20:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:52 dcaro: rebooting cloudcontrol1007 due to systemd-journal service failing to start
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-03 ===
* 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 16:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 12:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-02 ===
* 19:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-01 ===
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 14:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-06-27 ===
* 22:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet'
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1059.eqiad.wmnet'
* 19:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1059.eqiad.wmnet'
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1059.eqiad.wmnet'
* 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet'
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1064.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet'
* 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1067.eqiad.wmnet'
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet'
* 17:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1058.eqiad.wmnet'
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet'
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1066.eqiad.wmnet'
* 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet'
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1065.eqiad.wmnet'
* 17:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet'
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1057.eqiad.wmnet'
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 13:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 12:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet'
* 12:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirtlocal1001.eqiad.wmnet'
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 12:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 10:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 08:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 08:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 07:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 07:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2024-06-26 ===
* 22:32 andrewbogott: disabled all g3.* flavors in eqiad1
* 19:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 17:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 11:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 10:03 dcaro: taking cloudcephosd1006 out of the pool ([[phab:T348643|T348643]])
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 10:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-06-25 ===
* 22:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2003-dev.codfw.wmnet'
* 22:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2003-dev.codfw.wmnet'
* 22:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2002-dev.codfw.wmnet'
* 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2002-dev.codfw.wmnet'
* 22:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codfw.wmnet'
* 22:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2001-dev.codw.wmnet'
* 22:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codw.wmnet'
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2005-dev.codfw.wmnet'
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2004-dev.codfw.wmnet'
* 16:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt2006-dev.codfw.wmnet'
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2006-dev.codfw.wmnet'
* 03:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-24 ===
* 19:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet'
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056.eqiad.wmnet'
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet'
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet'
* 16:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet'
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet'

=== 2024-06-21 ===
* 10:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:35 wmbot~arturo@nostromo: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97)
* 10:35 wmbot~arturo@nostromo: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:35 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:35 wmbot~arturo@nostromo: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 09:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 08:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T368129|T368129]])
* 08:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T368129|T368129]])
* 04:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 4e612eb8-04e1-4541-941d-{{Gerrit|a05519eed60a}}
* 04:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 4e612eb8-04e1-4541-941d-{{Gerrit|a05519eed60a}}

=== 2024-06-20 ===
* 21:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 14789ac1-bc06-4677-9bb0-{{Gerrit|66c16c887427}}
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 14789ac1-bc06-4677-9bb0-{{Gerrit|66c16c887427}}
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet'
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet'
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet'
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1052.eqiad.wmnet'
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet'
* 13:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 614f9c99-86f1-410f-8ef5-{{Gerrit|e33d23215ff5}}
* 13:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 614f9c99-86f1-410f-8ef5-{{Gerrit|e33d23215ff5}}
* 13:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1051.eqiad.wmnet'
* 13:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet'
* 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1050.eqiad.wmnet'
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet'
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049.eqiad.wmnet'
* 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet'
* 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet'
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet'
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet'
* 10:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet'
* 09:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet'
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet'
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1045.eqiad.wmnet'
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 04:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server d0b1d9d5-1aec-4a05-a2d7-{{Gerrit|6d8522a365dc}}
* 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server d0b1d9d5-1aec-4a05-a2d7-{{Gerrit|6d8522a365dc}}
* 04:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 2d2e8925-b50f-483a-82ee-{{Gerrit|e6a1c588e5be}}
* 04:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 2d2e8925-b50f-483a-82ee-{{Gerrit|e6a1c588e5be}}
* 04:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 138be95d-93ad-4a85-9245-{{Gerrit|a0a508711555}}
* 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 138be95d-93ad-4a85-9245-{{Gerrit|a0a508711555}}
* 04:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server a3e945dc-3548-47fa-8ce3-{{Gerrit|bf1426ff3b15}}
* 04:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 0258b810-29af-448a-af5e-{{Gerrit|ed39e19286df}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server a3e945dc-3548-47fa-8ce3-{{Gerrit|bf1426ff3b15}}
* 04:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 37e23659-4516-4bd8-a9be-{{Gerrit|4cc55def5560}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 0258b810-29af-448a-af5e-{{Gerrit|ed39e19286df}}
* 04:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 7da824f3-c9f3-4460-b9b2-{{Gerrit|2a894268a7ca}}
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 37e23659-4516-4bd8-a9be-{{Gerrit|4cc55def5560}}
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 63b82d38-3026-408f-8dcc-{{Gerrit|0ecbd1a3c870}}
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 7da824f3-c9f3-4460-b9b2-{{Gerrit|2a894268a7ca}}
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 7cb371bb-a53a-4e65-a1cf-{{Gerrit|f1a8264a9166}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 63b82d38-3026-408f-8dcc-{{Gerrit|0ecbd1a3c870}}
* 04:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 6616fcbf-a49e-4e03-b735-{{Gerrit|84d31b535c08}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 6616fcbf-a49e-4e03-b735-{{Gerrit|84d31b535c08}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 7cb371bb-a53a-4e65-a1cf-{{Gerrit|f1a8264a9166}}
* 04:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 6f1171db-9d7d-466f-aaa7-{{Gerrit|cb14e1a6af41}}
* 04:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 6f1171db-9d7d-466f-aaa7-{{Gerrit|cb14e1a6af41}}
* 04:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 1dc3fcee-cf27-4351-ad60-{{Gerrit|384478624ea3}}
* 04:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 1dc3fcee-cf27-4351-ad60-{{Gerrit|384478624ea3}}
* 04:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 3df62375-e75d-4068-9c71-{{Gerrit|13519b6bf927}}
* 04:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 3af9b40f-d29d-4216-9b64-{{Gerrit|0ebb10f94c7c}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 3df62375-e75d-4068-9c71-{{Gerrit|13519b6bf927}}
* 04:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server ae9fa949-e4ce-4ffb-ad9f-{{Gerrit|4e5a3812d031}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server ae9fa949-e4ce-4ffb-ad9f-{{Gerrit|4e5a3812d031}}
* 04:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 452dc8d3-6ee0-412c-90ba-{{Gerrit|66f21f9d09c1}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 3af9b40f-d29d-4216-9b64-{{Gerrit|0ebb10f94c7c}}
* 04:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 452dc8d3-6ee0-412c-90ba-{{Gerrit|66f21f9d09c1}}
* 04:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 47b0da1d-e50a-42c1-8cd9-{{Gerrit|dc255cc2f1a3}}
* 04:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 47b0da1d-e50a-42c1-8cd9-{{Gerrit|dc255cc2f1a3}}
* 02:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T368007|T368007]])
* 02:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T368007|T368007]])
* 02:05 andrewbogott: cloudvirt1063 is unresponsive, cycling power from racadm

=== 2024-06-19 ===
* 15:43 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1044.eqiad.wmnet'
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 15:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet'
* 12:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 12:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet'
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet'
* 11:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet'
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1042.eqiad.wmnet'
* 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server f667d3c2-379a-48d7-ad44-{{Gerrit|4f3933bdb871}}
* 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server f667d3c2-379a-48d7-ad44-{{Gerrit|4f3933bdb871}}
* 01:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 71e4296d-039d-4452-9d92-{{Gerrit|69b9f8eb3aba}}
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 71e4296d-039d-4452-9d92-{{Gerrit|69b9f8eb3aba}}
* 01:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server a7725cd2-6162-41a3-8add-{{Gerrit|4dc0668b233b}}
* 01:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server a7725cd2-6162-41a3-8add-{{Gerrit|4dc0668b233b}}
* 01:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}

=== 2024-06-18 ===
* 21:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2024-06-17 ===
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T364457|T364457]])
* 20:03 andrewbogott: repaced ovs hosts in the 'ceph' aggregate
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T364457|T364457]])
* 18:16 andrewbogott: temporarily removing all ovs hosts from the 'ceph' aggregate so the scheduler will stop putting linuxbridge hosts on ovs hosts and breaking them
* 17:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T364457|T364457]])
* 13:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet'

=== 2024-06-14 ===
* 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet'
* 13:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet'

=== 2024-06-13 ===
* 16:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet'
* 13:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet'
* 13:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2003-dev.codfw.wmnet'
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2003-dev.codfw.wmnet'

=== 2024-06-12 ===
* 11:49 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1031.eqiad.wmnet'
* 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet'
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet'
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet'
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet'
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet'

=== 2024-06-11 ===
* 15:55 dcaro: restarting mon service on cloudcephmon1002 to try to release the 2 stuck ops left
* 13:30 taavi: pin all existing eqiad1 flavors to linuxbridge hypervisors [[phab:T364458|T364458]]
* 12:28 taavi: add all existing eqiad1 cloudvirts to new network-linuxbridge aggregate [[phab:T364458|T364458]]
* 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-07 ===
* 19:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-05 ===
* 11:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-06-04 ===
* 15:49 taavi: drop hopefully-unused 68.10.in-addr.arpa. from designate [[phab:T361220|T361220]]

=== 2024-05-30 ===
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-29 ===
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T364984|T364984]])
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T364984|T364984]])
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T364984|T364984]])
* 18:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T364984|T364984]])

=== 2024-05-28 ===
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-21 ===
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 08:18 taavi: stop neutron services on cloudnet1005 [[phab:T364459|T364459]]

=== 2024-05-20 ===
* 14:46 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:46 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:20 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:19 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:18 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:18 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:17 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:17 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:16 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:16 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:15 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:15 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs

=== 2024-05-16 ===
* 08:55 taavi: delete 'monitoring' project https://phabricator.wikimedia.org/T365105

=== 2024-05-15 ===
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T319184|T319184]])
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-05-14 ===
* 19:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-10 ===
* 13:23 andrewbogott: deploying updated 2024.1 Horizon

=== 2024-05-09 ===
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-25 ===
* 21:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T356287|T356287]])
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T356287|T356287]])
* 21:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1033' ([[phab:T356287|T356287]])
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033' ([[phab:T356287|T356287]])
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T356287|T356287]])
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T356287|T356287]])

=== 2024-04-23 ===
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2002-dev.codfw.wmnet'
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2002-dev.codfw.wmnet'

=== 2024-04-18 ===
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2008-dev.codfw.wmnet'
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2008-dev.codfw.wmnet'

=== 2024-04-17 ===
* 14:12 dcaro: deleting dns leaks
* 01:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])

=== 2024-04-16 ===
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 20:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2001-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2001-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T356287|T356287]])

=== 2024-04-15 ===
* 11:19 taavi: update spicerack to 8.5.0 on cloudcumin2001

=== 2024-04-10 ===
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-09 ===
* 18:13 andrewbogott: rebooting cloudinfra-cloudvps-puppetserver-1; unresponsive
* 13:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-05 ===
* 14:37 taavi: run maintain-replica-indexes on all web replicas [[phab:T361945|T361945]]
* 14:27 taavi: run maintain-replica-indexes on remaining analytics replicas [[phab:T361945|T361945]]
* 14:17 taavi: run maintain-replica-indexes on clouddb1017 [[phab:T361945|T361945]]

=== 2024-04-04 ===
* 18:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-03 ===
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T319184|T319184]])
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T319184|T319184]])
* 10:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T319184|T319184]])
* 10:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T319184|T319184]])
* 09:17 taavi: manually delete prometheus-node-textfile-wmcs-dnsleaks.service and related files from cloudservices1005/6, leftovers of the designate api to cloudcontrol migration
* 09:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T319184|T319184]])
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2024-04-02 ===
* 15:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T319184|T319184]])
* 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T319184|T319184]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 12:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 12:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-03-27 ===
* 11:47 taavi: deleting about 2k stale puppet certs by running wmcs-puppetcertleaks in delete mode

=== 2024-03-22 ===
* 10:25 dcaro: adding back cloudcephosd1034 to the pool after doing the performance tests ([[phab:T348643|T348643]])

=== 2024-03-21 ===
* 22:07 andrewbogott: doing dist-upgrade on cloudcontrol nodes to get mariadb upgraded for [[phab:T357133|T357133]]
* 22:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T357133|T357133]])
* 22:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T357133|T357133]])
* 11:08 dcaro: restarting nova-api on cloudcontrol1007

=== 2024-03-20 ===
* 17:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:27 dcaro: turning off cloudcephosd1030 to swap some disks ([[phab:T348643|T348643]])
* 15:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T348643|T348643]])
* 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2024-03-19 ===
* 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 13:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2024-03-18 ===
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-03-09 ===
* 16:43 andrewbogott: restarted nova-api on cloudcontrol1006

=== 2024-03-08 ===
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:21 arturo: restarted nova-api in cloudcontrol1007, it was complaining about mysql broken pipe
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 10:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 10:35 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2024-03-07 ===
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance

=== 2024-03-06 ===
* 17:46 dhinus: running &quot;wmcs-dnsleaks --delete&quot; to clean up 2 leaked records (tools-sgeweblight-10-32)
* 15:36 dcaro: renewing puppet ca cert for cloud-puppetmaster-03
* 15:24 dcaro: renewing puppet ca cert for cloudinfra-internal puppetmaster

=== 2024-03-04 ===
* 14:56 dhinus: delete project &quot;loggerdiscordbot&quot; in favor of new project &quot;discordbots&quot; [[phab:T358337|T358337]],[[phab:T358427|T358427]]
* 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reboot_node (exit_code=0) ([[phab:T359049|T359049]])
* 12:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reboot_node ([[phab:T359049|T359049]])

=== 2024-03-01 ===
* 14:59 taavi: removing wmf-auto-restart-cron from all VMs without cron via cumin - https://gerrit.wikimedia.org/r/c/operations/puppet/+/1007328/ [[phab:T358343|T358343]]
* 12:07 dcaro: restarted nova-api on cloudcontrol100* as it was very slow
* 12:04 dcaro: restarted nova-api on cloudcontrol1005 as it was very slow

=== 2024-02-27 ===
* 18:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-26 ===
* 10:02 arturo: deleting nskaggs account from gerrit's wmcs-trusted group

=== 2024-02-22 ===
* 13:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-21 ===
* 13:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) ([[phab:T319184|T319184]])
* 13:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance ([[phab:T319184|T319184]])
* 13:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T319184|T319184]])
* 13:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T319184|T319184]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-02-20 ===
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:45 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=0) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])

=== 2024-02-19 ===
* 12:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 10:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=0) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=99) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])

=== 2024-02-15 ===
* 17:34 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.roll_reboot_cloudnets ([[phab:T356975|T356975]])
* 15:34 taavi: restart radosgw in eqiad as I am seeing 500 errors
* 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:58 dhinus: restore correct aggregate &quot;localdisk&quot; for cloudvirtlocal1001 and remove &quot;maintenance&quot;
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T319184|T319184]])
* 05:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet[B&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet[B&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2024-02-14 ===
* 22:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtXXXX.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtXXXX.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:49 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:49 wmbot~andrew@bullseye: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97)
* 19:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:48 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:45 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:45 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:38 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:38 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:36 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:36 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:34 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:34 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:32 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 17:13 wmbot~fran@wmf3169: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtlocal1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T356975|T356975]])
* 17:12 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtlocal1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T356975|T356975]])
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:09 taavi: creating some missing $PROJECT.wmcloud.org. DNS zones
* 14:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:21 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 09:31 taavi: failover all dumps traffic to clouddumps1001
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 08:16 taavi: reboot clouddumps1001 for kernel updates

=== 2024-02-13 ===
* 17:07 wmbot~taavi@runko: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:07 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:06 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:06 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::eqiad1::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::eqiad1::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P:openstack::eqiad1::nova::compute::service'
* 16:02 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P:openstack::eqiad1::nova::compute::service'
* 16:01 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:01 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:10 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) ([[phab:T356975|T356975]])
* 14:55 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T356975|T356975]])

=== 2024-02-12 ===
* 17:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0)
* 17:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0)
* 17:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:23 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99)
* 17:21 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 17:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 17:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:36 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 16:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 15:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 15:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99)
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 01:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:34 andrewbogott: resetting eqiad1 rabbitmq in hopes of resolving neutron double message warnings
* 01:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-11 ===
* 21:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:57 andrewbogott: running wmcs.openstack.restart_openstack for all eqiad1 services
* 20:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 20:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-08 ===
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:43 taavi: deploy change to exclude cloud-private networks from general egress NAT https://phabricator.wikimedia.org/T356850

=== 2024-02-06 ===
* 20:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:09 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 02:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-05 ===
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:55 andrewbogott: rebuilt bookworm base image in eqiad1 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/992677

=== 2024-02-02 ===
* 13:54 arturo: [codfw1dev] cleanup /etc/network/interfaces on cloudlb2003-dev from puppet leftovers

=== 2024-02-01 ===
* 10:55 taavi: invite aborrero to /repos/cloud, /toolforge-repos, /cloudvps-repos on gitlab

=== 2024-01-29 ===
* 12:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-26 ===
* 09:01 taavi: joining cloudrabbit1001/2 to the cluster on 1003 [[phab:T345610|T345610]]

=== 2024-01-25 ===
* 16:48 andrewbogott: taavi just moved all rabbitmq traffic to cloudrabbit1003 as part of [[phab:T345610|T345610]]
* 16:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:27 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:27 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-2.tools.eqiad1.wikimedia.cloud to the cluster
* 13:15 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:15 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-1.tools.eqiad1.wikimedia.cloud to the cluster
* 12:48 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 12:48 wmbot~taavi@runko: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-1.toolsbeta.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-24 ===
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-22 ===
* 16:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:56 andrewbogott: restarting openstack sevices on eqiad1 to clean up from the mariadb restarts
* 15:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:07 andrewbogott: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/992192 and resetting galera cluster in eqiad1

=== 2024-01-21 ===
* 05:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 05:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-19 ===
* 23:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-18 ===
* 20:10 taavi: mysql:labsdbaccounts@m5-master.eqiad.wmnet [labsdbaccounts]&gt; update account_host set status = 'absent' where id = 137613;
* 12:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 12:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-101.tools.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-17 ===
* 15:17 andrewbogott: &quot;systemctl restart mariadb@s4.service mariadb@s6.service&quot; on clouddb1015. System is in danger of oom and there are no obvious long queries running

=== 2024-01-16 ===
* 14:25 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:25 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 13:55 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:55 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:54 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:54 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:53 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 13:53 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:52 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:52 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:35 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 11:34 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 11:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 09:41 taavi: drop dbproxy1018/9 grants from all clouddb hosts [[phab:T346947|T346947]]
* 09:24 taavi: move cloudvirt2004-dev from 'failed' to 'active' in netbox - seems like that was for [[phab:T348531|T348531]] which is now resolved

=== 2024-01-15 ===
* 14:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 14:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 14:32 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 12:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-11 ===
* 10:51 dcaro: restarting striker.service on cloudweb1003 as it seems non-responsive

=== 2024-01-10 ===
* 17:43 bd808: Blocking Developer accounts connected to invalid/legacy wikimedia.org email addresses ([[phab:T218239|T218239]])
* 16:48 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0) ([[phab:T346631|T346631]])
* 16:48 wmbot~fran@wmf3169: test message3 from local cookbook ([[phab:T346631|T346631]])
* 16:48 wmbot~fran@wmf3169: START - Cookbook wmcs.do_log_msg ([[phab:T346631|T346631]])

=== 2024-01-09 ===
* 17:50 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0) ([[phab:T346631|T346631]])
* 17:49 wmbot~fran@wmf3169: test message2 from local cookbook ([[phab:T346631|T346631]])
* 17:49 wmbot~fran@wmf3169: START - Cookbook wmcs.do_log_msg ([[phab:T346631|T346631]])
* 17:43 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])
* 17:43 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])
* 17:42 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])

=== 2024-01-08 ===
* 15:52 taavi: verify wmcloud.org, wmflabs.org and toolforge.org in gmail postmaster console to figure out how much google likes us ([[phab:T354112|T354112]])

=== 2024-01-07 ===
* 19:34 andrewbogott: removed cloudvirt1063 from 'ceph' aggregate, added to 'maintenance' aggregate   [[phab:T353408|T353408]]
* 19:34 andrewbogott: evacuating all VMs from cloudvirt1063.  [[phab:T353408|T353408]]

=== 2024-01-02 ===
* 16:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T353408|T353408]])
* 16:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T353408|T353408]])
* 10:22 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:22 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-12-31 ===
* 21:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:35 andrewbogott: running openstack service restart cookbook in eqiad1 in response to a bunch of service down alerts
* 21:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-21 ===
* 16:52 dhinus: puppet node deactivate cloudvirt1063.eqiad.wmnet [[phab:T353406|T353406]]
* 03:01 andrewbogott: restarting mariadb on cloudcontrol1005, hoping to get Galera back in sync

=== 2023-12-20 ===
* 19:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-18 ===
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:23 andrewbogott: restarting all eqiad1 openstack services after a rabbitmq upgrade/rebuild for [[phab:T353646|T353646]]
* 15:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-15 ===
* 13:13 dcaro: restarted nova-fullstack on codfw as it was stuck (and alerting through stale prometheus file)

=== 2023-12-14 ===
* 00:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 00:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 00:25 andrewbogott: evacuating hosts from cloudvirt1063 and depooling. [[phab:T353406|T353406]]

=== 2023-12-13 ===
* 16:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=99)

=== 2023-12-12 ===
* 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 17:45 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-100.tools.eqiad1.wikimedia.cloud to the cluster
* 16:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:11 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-99.tools.eqiad1.wikimedia.cloud to the cluster
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:49 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-98.tools.eqiad1.wikimedia.cloud to the cluster

=== 2023-12-10 ===
* 18:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:27 andrewbogott: restarting all openstack API servers, hoping to make things a bit more responsive
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-08 ===
* 12:00 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99)  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud ([[phab:T353055|T353055]])
* 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud ([[phab:T353055|T353055]])
* 11:58 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=97)  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud
* 11:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud
* 09:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) ([[phab:T345084|T345084]])
* 09:32 dcaro: restarting nova and keystone as they are getting too slow ([[phab:T345084|T345084]])
* 09:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack ([[phab:T345084|T345084]])
* 09:32 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) ([[phab:T345084|T345084]])
* 09:31 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack ([[phab:T345084|T345084]])

=== 2023-12-07 ===
* 13:12 dcaro: rebooting cloudcephosd1001 to make sure puppet7 migration went ok

=== 2023-12-04 ===
* 00:08 andrewbogott: rebooting cloudcontrol1006 to recover from full disk error

=== 2023-12-03 ===
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-02 ===
* 12:27 taavi: powercycle cloudvirt1063 [[phab:T352595|T352595]]
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:28 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-97.tools.eqiad1.wikimedia.cloud to the cluster
* 11:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-96.tools.eqiad1.wikimedia.cloud to the cluster
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 10:50 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-95.tools.eqiad1.wikimedia.cloud to the cluster
* 00:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:21 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-94.tools.eqiad1.wikimedia.cloud to the cluster
* 00:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-93.tools.eqiad1.wikimedia.cloud to the cluster
* 00:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-92.tools.eqiad1.wikimedia.cloud to the cluster
* 00:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-91.tools.eqiad1.wikimedia.cloud to the cluster
* 00:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:05 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-90.tools.eqiad1.wikimedia.cloud to the cluster
* 00:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:01 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-89.tools.eqiad1.wikimedia.cloud to the cluster

=== 2023-12-01 ===
* 17:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:19 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T351171|T351171]])
* 16:19 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 15:49 andrewbogott: reimaging cloudcontrol1005 due to widespread misbehavior
* 14:24 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T351171|T351171]])
* 14:20 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 14:19 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97) ([[phab:T351171|T351171]])
* 14:18 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 13:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:55 taavi: restart neutron-rpc-server.service on eqiad1 cloudcontrols

=== 2023-11-30 ===
* 20:44 andrewbogott: generating to application credentials for the tests that run on tf-infra-test
* 19:54 andrewbogott: reimaged cloudrabbit100[23] after https://gerrit.wikimedia.org/r/c/operations/puppet/+/979127.  I didn't reimage 1001 because that will require rebuilding the whole cluster but I did remove the related packages.
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet'
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet'
* 17:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet'
* 17:47 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet'
* 17:42 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:30 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:19 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:19 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:09 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:59 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:54 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:39 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:34 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:15 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:10 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:05 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:00 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:31 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:22 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:17 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:17 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:13 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:07 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:38 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:16 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:03 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 13:55 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 13:43 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T348843|T348843]])
* 13:43 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T348843|T348843]])
* 12:18 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 12:04 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:59 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:45 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:44 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:28 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T348843|T348843]])

=== 2023-11-29 ===
* 15:27 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:15 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:59 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T348843|T348843]])
* 14:50 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T348843|T348843]])

=== 2023-11-28 ===
* 14:18 taavi: moving wiki replica DNS to use cloudlbs instead of the old proxy VMs [[phab:T346947|T346947]]

=== 2023-11-27 ===
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-24 ===
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-22 ===
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 13:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-21 ===
* 10:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-20 ===
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-17 ===
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-16 ===
* 12:09 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:05 taavi@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 11:23 dhinus: upgraded spicerack from 8.0.2 to 8.0.3 on cloudcumins
* 05:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 05:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-15 ===
* 09:50 taavi: move cloudlb hosts to use the nftables firewall backend

=== 2023-11-14 ===
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 12:03 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T345811|T345811]])
* 11:44 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T345811|T345811]])
* 10:10 taavi: restart kiwix-mirror-update on clouddumps1001
* 05:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 05:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 04:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 04:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 04:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 04:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 03:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 03:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 03:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 03:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])

=== 2023-11-13 ===
* 22:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 22:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 22:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 22:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 21:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 19:31 andrewbogott: rebooting cloudcontrol2005-dev, trying to fix general misbehavior
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 19:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1033.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1032.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet'
* 17:09 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T345811|T345811]])
* 17:09 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T345811|T345811]])
* 16:57 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T345811|T345811]])
* 15:37 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T345811|T345811]])
* 15:19 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T345811|T345811]])
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-11 ===
* 02:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 00:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1058'
* 00:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1058'
* 00:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 00:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot

=== 2023-11-09 ===
* 21:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:54 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1025.eqiad.wmnet' ([[phab:T345811|T345811]])
* 14:52 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1025.eqiad.wmnet' ([[phab:T345811|T345811]])
* 14:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 14:50 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])

=== 2023-11-08 ===
* 16:43 andrewbogott: created foundationmemory project for [[phab:T350760|T350760]]

=== 2023-11-06 ===
* 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 15:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)

=== 2023-11-05 ===
* 00:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-11-04 ===
* 23:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 22:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 04:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)

=== 2023-11-03 ===
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:13 dhinus: triggering neutron failover from cloudnet1005 to cloudnet1006 ([[phab:T345811|T345811]])
* 06:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 01:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 01:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)

=== 2023-11-02 ===
* 20:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:32 taavi: merged cloudcontrol firewall cleanup patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/971211
* 16:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 16:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 13:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 11:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 11:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 07:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 07:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 05:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 03:47 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 03:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)

=== 2023-11-01 ===
* 23:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 21:29 taavi: re-enable puppet on cloudcontrol2006-dev.codfw.wmnet which has fallen off of puppetdb
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 20:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 19:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 14:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 09:04 taavi: reset local cookbook changes on cloudcumin1001 which were causing issues with puppet runs
* 09:02 taavi: restart nova-fullstack which had had some issues after yesterday's cloudcontrol1007 reimage
* 02:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 00:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-10-31 ===
* 23:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 18:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 14:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 10:16 dhinus: upgrading mariadb-server in cloudcontrol1005 ([[phab:T345811|T345811]])
* 10:04 dhinus: upgrading mariadb-server in cloudcontrol1006 ([[phab:T345811|T345811]])
* 09:51 dhinus: upgrading mariadb-server in cloudcontrol1007, second attempt ([[phab:T345811|T345811]])
* 06:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 05:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 02:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 02:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 02:27 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 02:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-10-30 ===
* 17:40 andrewbogott: rebooting tools-db-1.tools.eqiad1.wikimedia.cloud for yet another oom death
* 17:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 16:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 dhinus: upgrading mariadb-server in cloudcontrol1007 ([[phab:T345811|T345811]])
* 16:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-28 ===
* 07:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)

=== 2023-10-27 ===
* 15:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 15:38 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 15:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 15:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 15:29 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 15:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 15:27 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:09 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:05 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 09:04 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-26 ===
* 14:02 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 09:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-25 ===
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
* 11:00 taavi: update cloudcumins to spicerack 8.x
* 10:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster

=== 2023-10-24 ===
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-10-23 ===
* 16:48 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:06 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project catalyst in eqiad1
* 15:06 wm-bot2: fran@wmf3169 START - Cookbook wmcs.vps.create_project for project catalyst in eqiad1
* 10:36 taavi: merged change https://gerrit.wikimedia.org/r/c/operations/puppet/+/966494 which touches the pdns web server config

=== 2023-10-20 ===
* 15:17 dcaro: upgraded cloudcephosd1004 to v15 ([[phab:T349363|T349363]])
* 14:25 dcaro: upgraded cloudcephosd1003 to v15 ([[phab:T349363|T349363]])
* 13:20 dcaro: upgraded cloudcephosd1002 to v15 ([[phab:T349363|T349363]])
* 10:33 dcaro: upgraded cloudcephosd1001 to v15 ([[phab:T349363|T349363]])
* 08:26 dcaro: codfw ceph enabled diskprediction_local module, will take a bit to populate/start getting predictions ([[phab:T348716|T348716]])

=== 2023-10-17 ===
* 15:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T349109|T349109]])
* 15:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T349109|T349109]])

=== 2023-10-16 ===
* 03:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-10-13 ===
* 23:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 08:31 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 08:30 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 08:20 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])

=== 2023-10-12 ===
* 17:16 wm-bot2: fran@wmf3169 END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) ([[phab:T341285|T341285]])
* 17:16 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])

=== 2023-10-11 ===
* 10:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 10:36 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 07:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 06:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-10 ===
* 17:25 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:50 dcaro: removing ~100 dangling backup snapshots from eqiad1-compute ceph pool
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 12:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 11:38 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 11:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 11:33 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 11:32 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 11:00 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) ([[phab:T341285|T341285]])
* 11:00 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 10:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 09:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 09:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 09:43 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 08:19 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:18 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:18 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:17 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:17 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-09 ===
* 17:16 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:26 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:18 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:49 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:26 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 13:55 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 13:45 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 13:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:04 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:10 dcaro: undrained cephosd1011
* 09:09 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 09:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:06 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 09:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:05 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 09:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 07:35 taavi: restart postgresql on cloudbackup2001 [[phab:T348431|T348431]]

=== 2023-10-05 ===
* 16:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:29 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 16:24 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:11 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:30 arturo: operating on cloudgw @ eqiad1 ([[phab:T347469|T347469]])
* 14:07 wm-bot2: dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 12:55 arturo: doing cloudgw maintenance operations [[phab:T347469|T347469]]
* 11:54 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_rack
* 10:57 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 09:54 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 09:52 arturo: [codfw1dev] aborrero@cloudcontrol2001-dev:~ $ sudo keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
* 09:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 09:40 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 09:33 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 08:50 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 00:04 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)

=== 2023-10-04 ===
* 20:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 15:55 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 14:54 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 14:44 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 14:41 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 14:40 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 13:50 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:08 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:21 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:12 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-03 ===
* 19:38 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:35 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:32 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 11:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 08:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:43 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 08:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:39 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 08:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:23 dcaro: set .rgw.root pool on eqiad as rgw app (`ceph osd pool application enable .rgw.root rgw`)

=== 2023-10-02 ===
* 17:39 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:30 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:58 taavi: cloudcontrol1005,7: `sudo systemctl reset-failed keystone_sync_keys_from_cloudcontrol1006.eqiad.wmnet.service`
* 13:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:40 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:37 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 12:26 arturo: [codfw1dev] run `update domains set master = '185.15.57.25:5354 185.15.57.26:5354 172.20.5.9:5354 172.20.5.8:5354';` in cloudservies2005-dev
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])
* 11:55 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])
* 11:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 08:44 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])

=== 2023-09-29 ===
* 12:43 taavi: taavi@cloudcontrol1005 ~ $ os subnet set a69bdfad-d7d2-4cfa-8231-{{Gerrit|3d6d3e0074c9}} --no-dns-nameservers --dns-nameserver 172.20.255.1
* 08:36 taavi: start script to fix networking on broken bullseye instances [[phab:T347665|T347665]]

=== 2023-09-28 ===
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 12:01 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:48 arturo: rebooting cloudgw1001/1002 for sysctl and kernel upgrades

=== 2023-09-27 ===
* 12:07 arturo: merging cloudgw firewall changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/961360
* 09:36 taavi: move maintain-dbusers to cloudcontrol1005
* 01:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-26 ===
* 17:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-24 ===
* 15:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-23 ===
* 08:40 taavi: restart keystone

=== 2023-09-22 ===
* 14:03 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 14:02 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 14:02 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 14:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 14:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.undrain_node (exit_code=0)
* 14:00 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 14:00 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:56 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.undrain_node (exit_code=0)
* 13:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.undrain_node (exit_code=99)
* 13:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:48 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.undrain_node (exit_code=99)
* 13:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 13:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:46 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:44 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:41 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:37 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:33 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:33 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 11:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node

=== 2023-09-21 ===
* 16:35 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0)  on tools-db-3.tools.eqiad1.wikimedia.cloud
* 16:34 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs  on tools-db-3.tools.eqiad1.wikimedia.cloud
* 02:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-20 ===
* 21:49 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:45 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:38 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 21:35 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:31 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:12 arturo: moving openstack API endpoint to cloudlb ([[phab:T346439|T346439]])
* 10:28 arturo: running SQL command `update domains set master=&quot;172.20.1.5:5354 172.20.2.4:5354 185.15.56.162:5354 185.15.56.163:5354&quot;;` on cloudservices1005/1006 ([[phab:T346042|T346042]])
* 10:06 arturo: running SQL command `update domains set master=&quot;185.15.56.162:5354 185.15.56.163:5354&quot;` on cloudservices1005/1005 ([[phab:T346042|T346042]])

=== 2023-09-19 ===
* 18:54 andrewbogott: depooling clouddb1019 to let it recover from high memory use.  [[phab:T346826|T346826]]
* 15:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-18 ===
* 11:53 taavi: update designate urls in Keystone to point to openstack-next, until cloudlb is serving the main openstack address [[phab:T346042|T346042]]
* 11:40 arturo: decomission cloudservices1005 [[phab:T346042|T346042]] in preparation for re-racking
* 08:45 arturo: hardcode `185.15.56.161 openstack.eqiad1.wikimediacloud.org` in /etc/hosts in cloudcontrol1005 for [[phab:T346441|T346441]]

=== 2023-09-15 ===
* 11:43 arturo: merging NAT change for [[phab:T346426|T346426]] in cloudgw
* 10:33 arturo: faiolver cloudgw1001 into cloudgw1002, investigating a nftables syntax error ([[phab:T346432|T346432]])

=== 2023-09-14 ===
* 21:25 wm-bot2: andrew@bullseye END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:25 wm-bot2: andrew@bullseye START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:23 wm-bot2: andrew@bullseye END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:23 wm-bot2: andrew@bullseye START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 17:13 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 17:06 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 17:01 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:51 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:29 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:29 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:25 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) ([[phab:T345810|T345810]])
* 16:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:12 topranks: DNS operation: remove old DNS entry for ns0.openstack.eqiad1.wikimediacloud.org. on wikimedia authdns (was pointing to 208.80.154.148)
* 16:10 topranks: DNS operation: add new DNS entry for ns0.openstack.eqiad1.wikimediacloud.org. on wikimedia authdns pointing to 185.15.56.162
* 14:42 arturo: DNS operation: route 208.80.154.148 to cloudservices1006 in anticipation of cloudservices1005 decom ([[phab:T346042|T346042]])
* 12:11 arturo: enable puppet on cloudservices1006 to drop local NAT hacks and enable new DNS auth IP address ([[phab:T346042|T346042]])

=== 2023-09-13 ===
* 17:11 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) ([[phab:T345811|T345811]])
* 17:08 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 17:04 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 17:04 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:57 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:53 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:53 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:49 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:49 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 01:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 01:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 01:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-12 ===
* 18:06 andrewbogott: update domains set master='185.15.56.163:5354 208.80.154.11:5354 10.64.151.4:5354'; on cloudservices1005 + cloudservices1006
* 17:59 andrewbogott: mysql:root@localhost [pdns]&gt; update domains set master='185.15.56.163:5354 208.80.154.11:5354';
* 17:59 andrewbogott: &quot;designate-manage pool update' on cloudservices1005 to remove cloudservices1004 from the pool
* 17:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-11 ===
* 12:36 arturo: update DNS resolver cloud-wide to use 172.20.255.1 ([[phab:T342621|T342621]])

=== 2023-09-10 ===
* 02:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 02:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-07 ===
* 10:09 dhinus: reimaging cloudcontrol2001-dev to bookworm ([[phab:T345810|T345810]])

=== 2023-09-06 ===
* 16:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97)
* 16:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:53 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:37 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:37 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:37 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:35 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:35 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:34 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:34 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:23 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97)
* 16:23 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:13 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99)
* 16:12 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance

=== 2023-09-05 ===
* 12:34 arturo: synced pdns database from cloudservices1004 to cloudservices1006 ([[phab:T345240|T345240]])
* 12:18 arturo: updating pools.yaml in all cloudservices designate nodes ([[phab:T345240|T345240]])
* 10:54 arturo: running SQL command `update domains set master=&quot;208.80.154.11:5354 208.80.154.148:5354 10.64.151.4:5354&quot;;` on all 3 cloudservices nodes ([[phab:T345240|T345240]])

=== 2023-09-04 ===
* 15:58 arturo: stop and mask designate-sink.service @ cloudservices1006
* 14:19 arturo: started all designate services on cloudservices1006 [[phab:T345240|T345240]]
* 10:46 arturo: added designate galera DB grants for cloudlb [[phab:T345240|T345240]]
* 08:40 arturo: stopped all designate services on cloudservices1006 [[phab:T345240|T345240]]

=== 2023-09-01 ===
* 10:28 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.network.tests (exit_code=1) ([[phab:T345282|T345282]])
* 10:25 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.network.tests ([[phab:T345282|T345282]])

=== 2023-08-31 ===
* 15:14 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=99)
* 15:14 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 12:49 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudnet.show (exit_code=0)
* 12:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudnet.show
* 12:48 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 12:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 12:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 12:46 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:46 wm-bot2: fran@wmf3169 START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2023-08-30 ===
* 13:07 wm-bot2: dcaro testing stuff

=== 2023-08-28 ===
* 15:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:05 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-23 ===
* 16:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:10 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:08 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:08 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-15 ===
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:32 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 19:32 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:01 andrewbogott: rebooting cloudvirt2001-dev in an attempt to figure out what's happening with bastions
* 15:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:42 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:39 dcaro: removed some logs from the cloudmetrics1003:/var/log/carbon/ directory and stopped the carbon processes (they were crashing and filling up the disk with logs)

=== 2023-08-14 ===
* 22:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:11 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:08 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-07 ===
* 09:39 taavi: cloud vps graphite service was disabled: https://wikitech.wikimedia.org/wiki/News/2023_Cloud_VPS_metrics_changes [[phab:T326266|T326266]]

=== 2023-08-03 ===
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0)
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.do_log_msg
* 13:07 wm-bot2: Test SAL log ([[phab:T341793|T341793]]) - cookbook ran by root@cloudcumin1001

=== 2023-07-31 ===
* 16:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0)
* 16:16 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 16:15 fnegri@cloudcumin1001: START - Cookbook wmcs.do_log_msg
* 15:33 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:42 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:35 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 13:55 andrewbogott: recreating the codfw1dev galera cluster according to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Galera -- mariadb is stopped (and won't start) on all three cloudcontrol nodes
* 13:39 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001

=== 2023-07-27 ===
* 10:52 arturo: adding cloud-private subnet to cloudnet1005/1006 hosts in eqiad1 ([[phab:T342619|T342619]])

=== 2023-07-19 ===
* 14:02 wm-bot2: Draining cloudvirt2002-dev.codfw.wmnet ([[phab:T335840|T335840]]) - cookbook ran by raymond@ubuntu
* 14:02 wm-bot2: Safe rebooting cloudvirt2002-dev.codfw.wmnet ([[phab:T335840|T335840]]) - cookbook ran by raymond@ubuntu

=== 2023-07-17 ===
* 15:55 arturo: cloudcontrol1005 was shutdown earlier today ([[phab:T341495|T341495]])
* 15:35 arturo: [codfw1dev] cloudweb2002-dev up and running after reracking ([[phab:T327919|T327919]])
* 15:11 arturo: [codfw1dev] powered off cloudweb2002-dev for reracking ([[phab:T327919|T327919]])
* 12:45 taavi: removing diamond from remaining buster instances [[phab:T317032|T317032]]

=== 2023-07-13 ===
* 10:48 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:38 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:33 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay

=== 2023-07-07 ===
* 10:28 taavi: backfilling &lt;nowiki&gt;{&lt;/nowiki&gt;project&lt;nowiki&gt;}&lt;/nowiki&gt;.wmcloud.org and other currently-named DNS zones to projects that don't have them

=== 2023-07-06 ===
* 17:02 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:00 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:00 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-07-04 ===
* 14:08 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:07 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001

=== 2023-06-30 ===
* 21:37 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-06-29 ===
* 19:49 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:47 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-06-26 ===
* 08:46 arturo: [codfw1dev] manually start mariadb service @ cloudinfra-db-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud

=== 2023-06-24 ===
* 19:21 wm-bot2: Created new flavor: g3.cores16.ram34.disk20 (id:7dd33202-32c3-4bc7-b2d4-{{Gerrit|10c2ebe7e5c5}}) - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Created new flavor: g3.cores16.ram34816.disk20.admin (id:d76925a8-b58b-489e-a3d9-{{Gerrit|c68c7744b551}}) - cookbook ran by andrew@bullseye

=== 2023-06-23 ===
* 16:45 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 13:59 andrewbogott: rebooting every VM in codfw1dev

=== 2023-06-13 ===
* 15:35 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 14:58 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye

=== 2023-06-12 ===
* 19:37 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 11:57 arturo: [codfw1dev] refresh various occurrences of old FQDNs in instance puppet via horizon ([[phab:T324992|T324992]])

=== 2023-06-08 ===
* 16:56 andrewbogott: deleting all Stretch base images from glance
* 16:45 andrewbogott: updated the bullseye image with https://cloud.debian.org/images/cloud/bullseye/20230601-1398/debian-11-genericcloud-amd64-20230601-1398.tar.xz
* 12:17 wm-bot2: Drained cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:17 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 02920314-1efe-4934-ad81-{{Gerrit|d2a6cf2e17ab}}, use this to unset) ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:16 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:06 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 769349bf-465f-4f0c-a8f3-{{Gerrit|f2423631ba7e}}, use this to unset) ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus

=== 2023-06-07 ===
* 10:06 dcaro: upgraded ruby2.5 to latest fixed version on all buster VMs
* 08:10 dcaro: downgrading ruby2.5 to previous backport on all buster VMs

=== 2023-06-06 ===
* 19:09 andrewbogott: also increased RAM and secgroup-rule quota for Trove  [[phab:T337882|T337882]]
* 19:06 andrewbogott: increased trove secgroups, instances, volumes quotas from 40 to 100.  Trove is too popular!  [[phab:T337882|T337882]]
* 18:31 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 18:31 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:26 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:26 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:42 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:42 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye

=== 2023-06-05 ===
* 09:41 arturo: [codfw1dev] rebooting bastion-codfw1dev-02 (no IP address in the main interface) [[phab:T336963|T336963]]

=== 2023-06-02 ===
* 14:40 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by arturo@nostromo
* 12:59 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 12:58 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:58 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:44 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 12:44 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:43 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-26 ===
* 16:03 andrewbogott: &quot;maintain-views --all-databases --replace-all&quot; on clouddb1021 for [[phab:T337446|T337446]]

=== 2023-05-22 ===
* 19:54 andrewbogott: deleting project 'citelearn' as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge#SHUTDOWN_citelearn

=== 2023-05-21 ===
* 23:29 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 23:27 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:44 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:44 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-19 ===
* 14:53 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-18 ===
* 21:39 andrewbogott: deleting obsolete roles '4d8cad783d6342efa8414d7d36fbc034 {{!}} projectadmin_renamed_for_[[phab:T330759|T330759]]' and 'f473273fac7146b3bdbf22e5d4504f95 {{!}} user_renamed_for_[[phab:T330759|T330759]]' on eqiad1. State pre-deletion is dumped to /root/allassignmentspredeletion.txt  on cloudcontrol1007.
* 21:35 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:33 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:18 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:10 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Restarting openstack services on cloudcontrol2001-dev@local1: ['cinder-volume'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:20 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-15 ===
* 14:28 wm-bot2: Drained cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:28 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: 96ce2ed0-3aff-4d04-be0b-{{Gerrit|e16513070617}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:27 wm-bot2: Draining cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:23 wm-bot2: Drained cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 110176f8-04d5-4110-bb7d-{{Gerrit|1ab272bd8be2}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: c6e92e13-49f4-4db3-8a13-{{Gerrit|8692ccfd3bc9}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:12 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:12 wm-bot2: Drained cloudvirt1035.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:11 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: fa730dec-848f-45fb-9eda-{{Gerrit|e74bd874c5c9}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:10 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1006: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1007: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1005: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:01 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: eb1cfac0-d481-4baa-b9cd-{{Gerrit|15e5fbcef495}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:00 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:58 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: 3e6c3ff3-7d55-4777-9032-{{Gerrit|b867a257eced}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:57 wm-bot2: Draining cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:53 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: 0693664a-df78-417e-ba34-{{Gerrit|590e5a0a9981}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:52 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:49 wm-bot2: Set cloudvirt cloudvirt1035.eqiad.wmnet maintenance (downtime id: e6929ab8-4bc3-4186-817b-{{Gerrit|9b53dbd597c6}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:48 wm-bot2: Draining cloudvirt1035.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 andrewbogott: restarting nova services in eqiad1, trying to free up db connections
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:33 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: fef43d73-6fd4-4dde-a0ac-{{Gerrit|95fd69a9b0c1}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:32 wm-bot2: Draining cloudvirt1034.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 5f867662-e824-498c-a715-{{Gerrit|2e2ad50f0bb5}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: 0f515150-1313-41d4-a5f6-{{Gerrit|9bc00ce9b245}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Draining cloudvirt1033.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Drained cloudvirt1032.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot2: Set cloudvirt cloudvirt1032.eqiad.wmnet maintenance (downtime id: e826adc3-addd-44d8-b39e-{{Gerrit|ae7bd2df1e60}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:49 wm-bot2: Draining cloudvirt1032.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:49 wm-bot2: Drained cloudvirt1031.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:14 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 4154c818-744c-4d84-9883-{{Gerrit|cae7a5826ed5}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:13 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:13 wm-bot2: Drained cloudvirt1026.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:04 wm-bot2: Set cloudvirt cloudvirt1026.eqiad.wmnet maintenance (downtime id: cdfc3d01-ec1e-483c-9a38-{{Gerrit|834193e487ff}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Draining cloudvirt1026.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Drained cloudvirt1025.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 11:51 wm-bot2: Set cloudvirt cloudvirt1025.eqiad.wmnet maintenance (downtime id: 6a56757d-35de-499e-8209-{{Gerrit|728bcf62a22a}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 11:50 wm-bot2: Draining cloudvirt1025.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-12 ===
* 17:52 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-10 ===
* 16:09 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 andrewbogott: running &quot;cookbook -c ~/.config/spicerack/cookbook_config.yaml wmcs.openstack.restart_openstack --cluster-name eqiad1 --all&quot; to pick up changes for testing [[phab:T336379|T336379]]
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 03:05 andrewbogott: &quot;systemctl restart puppet-enc&quot; on enc-1.cloudinfra.eqiad1.wikimedia.cloud.  Seems to have crashed.

=== 2023-05-05 ===
* 16:07 wm-bot2: Drained cloudvirt1024.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:03 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 95995009-09d6-496e-8cd2-{{Gerrit|0cfac93d3cf7}}, use this to unset) ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:02 wm-bot2: Draining cloudvirt1024.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Drained cloudvirt1023.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:51 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 53c46cae-00af-4664-97ff-{{Gerrit|266b393335bb}}, use this to unset) ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:50 wm-bot2: Draining cloudvirt1023.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:49 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 528ea4f6-8088-475e-937f-{{Gerrit|098ffba861b6}}, use this to unset) - cookbook ran by andrew@bullseye
* 15:47 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 3ef85b5e-d9d9-4b24-901b-{{Gerrit|a3058a7d0615}}, use this to unset) - cookbook ran by andrew@bullseye
* 15:44 andrewbogott: moved cloudvirt1023 and cloudvirt1024 from 'ceph' aggregate to 'maintenance' aggregate, prep for decom [[phab:T336064|T336064]]
* 15:44 andrewbogott: moved cloudvirt1028 from 'localdisk' aggregate to 'maintenance' aggregate. Nothing new should be scheduled here, local storage should now move to cloudvirtlocal100x
* 15:41 andrewbogott: moved cloudvirt1055 and cloudvirt1056 from 'spare' to 'ceph' aggregate.  Prep for removing two obsolete cloudvirts, 1023 and 1024.  [[phab:T336064|T336064]]

=== 2023-05-04 ===
* 22:49 andrewbogott: removed fullstack-* puppet reports on puppetmaster-02.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud and cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs to free up disk space

=== 2023-05-02 ===
* 13:01 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:31 wm-bot2: Destroying OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 12:30 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Added 1 new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Added OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus

=== 2023-05-01 ===
* 17:09 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 17:09 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 17:08 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 17:08 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 15:22 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 13:53 taavi: running wmcs-novastats-puppetleaks in real mode [[phab:T334127|T334127]]
* 13:51 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus

=== 2023-04-18 ===
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1007: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1006: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1005: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudvirt1056: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1019: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1060: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1023: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1038: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1036: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1039: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1050: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1061: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1028: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1020: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1052: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1040: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1043: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1026: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1030: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1048: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1025: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1035: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1055: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1042: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1059: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1057: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1032: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1029: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1044: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1047: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1034: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1058: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1045: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1041: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1031: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1046: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1027: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1054: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1033: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1037: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1051: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1053: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1049: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1019: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1020: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:41 andrewbogott: resetting rabbitmq on cloudrabbit1003 due to splitbrain
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1019: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1020: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye

=== 2023-04-17 ===
* 08:49 wm-bot2: Increased quotas by 9 cores, 1 instances, 16 ram ([[phab:T334695|T334695]]) - cookbook ran by dcaro@vulcanus

=== 2023-04-06 ===
* 17:03 andrewbogott: running wmcs-wikireplica-dns on cloudcontrol1005 to update tools-db dns entries

=== 2023-04-04 ===
* 17:23 andrewbogott: resetting all three rabbitmq nodes and restarting all openstack services as per https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Rabbitmq#Resetting_the_HA_setup

=== 2023-03-28 ===
* 13:56 andrewbogott: depooling cloudweb1003 before switch upgrade
* 10:58 dhinus: disabled tool &quot;wb&quot; by clicking the disable button at https://toolsadmin.wikimedia.org/tools/id/wb [[phab:T328693|T328693]]
* 08:34 arturo: cleanup neutron agents for cloudvirt1021/1022 (decom)
* 08:32 arturo: cleanup neutron agents for cloudvirt1017 (decom)

=== 2023-03-27 ===
* 14:30 wm-bot2: Drained cloudvirt1024.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:19 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 3f43d3ca-696c-4d3c-8d5b-{{Gerrit|e57984f0eb86}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:18 wm-bot2: Draining cloudvirt1024.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:17 wm-bot2: Drained cloudvirt1023.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:08 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: f4767781-ea26-453b-9521-{{Gerrit|847a8340a249}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:07 wm-bot2: Draining cloudvirt1023.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:04 wm-bot2: Drained cloudvirt1022.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:55 wm-bot2: Set cloudvirt cloudvirt1022.eqiad.wmnet maintenance (downtime id: 0e794cfe-5896-46c1-842d-{{Gerrit|c34719140d4f}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:54 wm-bot2: Draining cloudvirt1022.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:54 wm-bot2: Drained cloudvirt1021.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:48 wm-bot2: Set cloudvirt cloudvirt1021.eqiad.wmnet maintenance (downtime id: e7a904ca-003d-450a-ad85-{{Gerrit|886fb80dfc41}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:47 wm-bot2: Draining cloudvirt1021.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:46 wm-bot2: Drained cloudvirt1017.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Set cloudvirt cloudvirt1017.eqiad.wmnet maintenance (downtime id: 0ff0090f-26c3-4278-a9c9-{{Gerrit|cce518559408}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:35 wm-bot2: Draining cloudvirt1017.eqiad.wmnet - cookbook ran by andrew@bullseye

=== 2023-03-22 ===
* 12:41 taavi: delete wmde-templates-alpha project [[phab:T332773|T332773]]

=== 2023-03-08 ===
* 21:45 bd808: maintain-kubeusers container in CrashLoopBackoff, investigating
* 13:49 dcaro: stopping puppet on labostre1004 to debug maintain-dbusers

=== 2023-03-07 ===
* 16:06 andrewbogott: updated application credential roles, replacing 'user' with 'reader' and 'projectadmin' with 'member':  update application_credential_role set role_id='f75a3c410bca4e96a1cf6ac103b0ccaf' where role_id='f473273fac7146b3bdbf22e5d4504f95'  and update application_credential_role set role_id='38676f30eaeb44518bf7e144a73c8da6' where role_id='4d8cad783d6342efa8414d7d36fbc034'
* 10:11 dcaro: there was a little unavailability for some VMs while ceph was starting to rebalance things, but it seems stable and moving data around ([[phab:T331141|T331141]])
* 09:37 dcaro: Changing ceph crush map to allow rack HA on eqiad1 cluster ([[phab:T331141|T331141]])

=== 2023-03-03 ===
* 12:12 arturo: installing haproxy updates ([[phab:T331119|T331119]])

=== 2023-03-02 ===
* 16:17 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1010.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Added 1 new OSDs ['cloudcephosd1010.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Added OSD cloudcephosd1010.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot2: Finished rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:10 wm-bot2: Rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Adding OSD cloudcephosd1010.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Adding new OSDs ['cloudcephosd1010.eqiad.wmnet'] to the cluster ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 10:43 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1005.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:44 wm-bot2: Added 1 new OSDs ['cloudcephosd1005.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:44 wm-bot2: Added OSD cloudcephosd1005.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:36 wm-bot2: Finished rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:33 wm-bot2: Rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Adding OSD cloudcephosd1005.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Adding new OSDs ['cloudcephosd1005.eqiad.wmnet'] to the cluster ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-28 ===
* 22:47 andrewbogott: adding new 'member' role assignment to every user/project pair that currently has the 'user' assignment.  [[phab:T330759|T330759]]
* 09:47 wm-bot2: Depooled and destroyed OSD daemons [79, 78, 77, 76, 75, 74, 73, 72] and removed the OSD host cloudcephosd1010 from the CRUSH map. ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:46 wm-bot2: Destroying OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:28 wm-bot2: Depooling OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot2: Depooling OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot2: Depooled and destroyed OSD daemons [39, 38, 37, 36, 35, 34, 33, 32] and removed the OSD host cloudcephosd1005 from the CRUSH map. ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:11 wm-bot2: Destroying OSDs with ids in [39, 38, 37, 36, 35, 34, 33, 32] on cloudcephosd1005 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:55 wm-bot2: Depooling OSDs with ids in [39, 38, 37, 36, 35, 34, 33, 32] on cloudcephosd1005 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-27 ===
* 21:01 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1004.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:53 wm-bot2: Added 1 new OSDs ['cloudcephosd1004.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:53 wm-bot2: Added OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:45 wm-bot2: Finished rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:42 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:41 wm-bot2: Adding OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:41 wm-bot2: Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Adding OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:14 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1003.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:13 wm-bot2: Added 1 new OSDs ['cloudcephosd1003.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:13 wm-bot2: Added OSD cloudcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:05 wm-bot2: Finished rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:02 wm-bot2: Rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding OSD cloudcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding new OSDs ['cloudcephosd1003.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:59 wm-bot2: Adding OSD coludcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:59 wm-bot2: Adding new OSDs ['coludcephosd1003.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:58 wm-bot2: Adding OSD coludcephosd1003... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:58 wm-bot2: Adding new OSDs ['coludcephosd1003'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-22 ===
* 08:25 wm-bot2: Depooled and destroyed OSD daemons [31, 30, 29, 28, 27, 26, 25, 24] and removed the OSD host cloudcephosd1004 from the CRUSH map. ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:22 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:15 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:13 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 07:23 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 07:02 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-21 ===
* 20:27 andrewbogott: deleted 200 more orphaned VM images with wmcs-novastats-cephleaks
* 17:18 andrewbogott: shutting down postgres on clouddb1004/1003, then shutting down the vms
* 15:50 wm-bot2: Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:48 wm-bot2: Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 14:00 wm-bot2: Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-16 ===
* 19:14 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1002.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:55 dcaro: Manually zapped /dev/sdc on cloudcephosd1002, probably a leftover drive since the beginning (or during the reimage the drives changed names, and this one had leftovers from the previous OS) ([[phab:T329498|T329498]])
* 17:47 wm-bot2: Added 1 new OSDs ['cloudcephosd1002.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:47 wm-bot2: Added OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:42 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:42 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:50 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:50 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:00 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:00 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot2: Added 1 new OSDs ['cloudcephosd1001.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Destroying OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot2: Depooling OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:20 wm-bot2: Destroying OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:19 wm-bot2: Depooling OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:03 wm-bot2: Destroying OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:01 wm-bot2: Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 10:59 wm-bot2: Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 10:15 dcaro: purges osd daemons 48 and 40 from eqiad ceph cluster ([[phab:T329709|T329709]])

=== 2023-02-15 ===
* 14:53 andrewbogott: deleting another 100 leaked VM images with wmcs-novastats-cephleaks
* 13:39 wm-bot2: Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:13 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:11 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:11 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot2: Destroying OSDs with id [[12345]] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:09 wm-bot2: Destroying OSDs with id ['12345'] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus

=== 2023-02-14 ===
* 13:17 andrewbogott: restarting all eqiad1 openstack services because that seems to sometimes help things *shrug*

=== 2023-02-13 ===
* 14:06 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 8fbf6bfd-eec1-4d81-8e0d-{{Gerrit|ea431d8411ee}} ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:32 taavi: re-enable puppet on labstore1004 [[phab:T329377|T329377]]

=== 2023-02-09 ===
* 21:17 andrewbogott: deleted 10% of leaked VM ceph images using wmcs-novastats-cephleaks (only 10% out of an abundance of caution)

=== 2023-02-08 ===
* 17:08 arturo: changing to cloudgw network setup, make VIPs /32 ([[phab:T295774|T295774]])

=== 2023-02-07 ===
* 11:26 arturo: [codfw1dev] testing network changes in cloudgw, expect unrealiable network ([[phab:T295774|T295774]])

=== 2023-02-04 ===
* 13:44 taavi: drop old columns from oathauth_users table on labtestwiki [[phab:T328131|T328131]]

=== 2023-02-03 ===
* 15:00 andrewbogott: restarted nova services in eqiad1 in an attempt to eke out another day or two of stability
* 14:13 taavi: attached GrapheSuppression developer account to wikitech

=== 2023-02-02 ===
* 13:14 dcaro_away: draining osd.48 from node cloudcephosd1001 ([[phab:T316544|T316544]])
* 12:57 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 7ac2b25a-d1bb-4789-8aa6-{{Gerrit|b9435b505349}} ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus

=== 2023-01-30 ===
* 22:34 wm-bot2: Upgraded and rebooted host cloudrabbit1002.wikimedia.org - cookbook ran by andrew@bullseye
* 21:34 andrewbogott: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/884922 and upgrading rabbitmq nodes for [[phab:T328155|T328155]]

=== 2023-01-27 ===
* 20:08 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:22 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 15:25 andrewbogott: restarting openstack services in eqiad1, another attempt to address instability

=== 2023-01-26 ===
* 20:34 andrewbogott: shutting down mariadb on cloudbackup2001-dev, testing the waters for [[phab:T328079|T328079]]

=== 2023-01-22 ===
* 03:42 andrewbogott: reset eqiad1 rabbitmq in an attempt to resolve some mild instability

=== 2023-01-20 ===
* 15:26 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 15:26 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['f47a3d91-b270-4c90-acc8-d85075a6bf8e'], use this to unset) - cookbook ran by andrew@bullseye
* 13:15 arturo: reinstall python3-neutron (to reset manual patching) on all cloudnet nodes and patch it via puppet, then restart neutron-l3-agent by hand ([[phab:T327463|T327463]])
* 10:12 arturo: [codfw1dev] failover neutron-l3-agent between cloudnet2005-dev/cloudnet2006-dev a couple of times [[phab:T327463|T327463]]
* 02:17 andrewbogott: stopping neutron-l3-agent on cloudnet1005 because it's logging at a furious rate and about to fill the drive

=== 2023-01-19 ===
* 18:06 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 18:06 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['36d5af6a-7d8e-4d0c-831e-1bf05c255984'], use this to unset) - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 17:22 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['66ec4f04-d25b-4067-be9e-2fe12cb1d3ff'], use this to unset) - cookbook ran by andrew@bullseye

=== 2023-01-18 ===
* 22:32 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 347cb75e-215e-4b85-ae14-{{Gerrit|4ce1934c70c7}}, use this to unset) - cookbook ran by andrew@bullseye
* 22:01 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 9bf6212f-7fdb-4869-8190-{{Gerrit|b07387b2bc7e}}, use this to unset) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 11aec8ea-f443-41ae-b79a-{{Gerrit|5e1e3aa94546}}, use this to unset) - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:03 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:38 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:38 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:37 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:37 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-01-17 ===
* 19:32 wm-bot2: Upgraded and rebooted host cloudbackup2002.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:04 wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye
* 17:52 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 17:23 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye

=== 2023-01-13 ===
* 17:36 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:26 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:26 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:25 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:21 wm-bot2: Restarting openstack services: &lt;nowiki&gt;{&lt;/nowiki&gt;'cloudcontrol2001-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'], 'cloudcontrol2004-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'], 'cloudvirt2001-dev': ['nova-compute'], 'cloudvirt2002-dev': ['nova-compute'], 'cloudvirt2003-dev': ['nova-compute'], 'cloudcontrol2005-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata
* 10:41 arturo: restart backup_vm.service on cloudbackup1003/1004 to recover from a nova Unknown Error (HTTP 503)

=== 2023-01-12 ===
* 22:34 andrewbogott: updated the Bullseye base image with the upstream {{Gerrit|20221219}} build
* 14:12 wm-bot2: Created project checkuser-beta-wiki with default quotas. ([[phab:T326740|T326740]]) - cookbook ran by arturo@nostromo

=== 2023-01-06 ===
* 18:42 wm-bot2: Safe reboot of cloudvirt2003-dev.codfw.wmnet finished successfully - cookbook ran by andrew@bullseye
* 18:42 wm-bot2: unset cloudvirt2003-dev.codfw.wmnet maintenance (aggregates: ceph) - cookbook ran by andrew@bullseye
* 18:39 wm-bot2: Drained cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: b50d9d3a-4f1d-4522-b86f-{{Gerrit|722fc9c55c87}}, use this to unset) - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Safe rebooting cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Draining cloudvirt2003-dev.cdofw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Safe rebooting cloudvirt2003-dev.cdofw.wmnet - cookbook ran by andrew@bullseye
* 00:14 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 00:08 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 00:01 wm-bot2: Upgraded and rebooted host cloudnet2006-dev.codfw.wmnet - cookbook ran by andrew@bullseye

=== 2023-01-05 ===
* 23:54 wm-bot2: Upgraded and rebooted host cloudnet2005-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 23:38 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 23:25 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 23:13 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 22:18 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 22:10 andrewbogott: upgrading codfw1dev openstack to version 'zed'

=== 2023-01-04 ===
* 21:18 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org - cookbook ran by andrew@bullseye
* 21:11 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org - cookbook ran by andrew@bullseye
* 20:12 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org - cookbook ran by andrew@bullseye
* 20:04 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org - cookbook ran by andrew@bullseye
* 14:45 wm-bot2: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] - cookbook ran by fran@wmf3169
* 14:44 wm-bot2: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:41 wm-bot2: Rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:41 wm-bot2: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:38 wm-bot2: Rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:37 wm-bot2: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev - cookbook ran by fran@wmf3169

=== 2023-01-03 ===
* 22:11 wm-bot2: Upgraded and rebooted host cloudservices2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Upgraded and rebooted host cloudservices2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 15:25 taavi: restart designate-sink everywhere to pick up wmf-sink changes

=== 2022-12-25 ===
* 14:21 taavi: register developer account 'instance-puppet-user-dev' to update the codfw1dev instance-puppet repo without access to the eqiad1 repo [[phab:T318504|T318504]]

=== 2022-12-22 ===
* 15:16 dcaro: added submit rights for JenkinsBot on all cloud/* gerrit repos

=== 2022-12-21 ===
* 04:59 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:59 wm-bot2: Finished rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:35 wm-bot2: Rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:35 wm-bot2: Finished rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:12 wm-bot2: Rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:12 wm-bot2: Finished rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:48 wm-bot2: Rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:48 wm-bot2: Finished rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:24 wm-bot2: Rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:24 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:00 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:00 wm-bot2: Finished rebooting node cloudcephosd1024.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:36 wm-bot2: Rebooting node cloudcephosd1024.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:36 wm-bot2: Finished rebooting node cloudcephosd1023.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:12 wm-bot2: Rebooting node cloudcephosd1023.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:12 wm-bot2: Finished rebooting node cloudcephosd1022.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:47 wm-bot2: Rebooting node cloudcephosd1022.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:47 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:22 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:22 wm-bot2: Finished rebooting node cloudcephosd1020.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:58 wm-bot2: Rebooting node cloudcephosd1020.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:58 wm-bot2: Finished rebooting node cloudcephosd1019.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:34 wm-bot2: Rebooting node cloudcephosd1019.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:34 wm-bot2: Finished rebooting node cloudcephosd1018.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:09 wm-bot2: Rebooting node cloudcephosd1018.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:09 wm-bot2: Finished rebooting node cloudcephosd1017.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye

=== 2022-12-20 ===
* 23:45 wm-bot2: Rebooting node cloudcephosd1017.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:45 wm-bot2: Finished rebooting node cloudcephosd1016.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:21 wm-bot2: Rebooting node cloudcephosd1016.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:21 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:56 wm-bot2: Rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:55 wm-bot2: Finished rebooting node cloudcephosd1014.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Rebooting node cloudcephosd1014.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Finished rebooting node cloudcephosd1013.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:08 andrewbogott: restarting openstack services with wmcs.openstack.restart_openstack due to miscellaneous trove failures
* 22:06 wm-bot2: Rebooting node cloudcephosd1013.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:06 wm-bot2: Finished rebooting node cloudcephosd1012.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Rebooting node cloudcephosd1012.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Finished rebooting node cloudcephosd1011.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Rebooting node cloudcephosd1011.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Finished rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Finished rebooting node cloudcephosd1009.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:26 wm-bot2: Rebooting node cloudcephosd1009.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:26 wm-bot2: Finished rebooting node cloudcephosd1008.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:01 wm-bot2: Rebooting node cloudcephosd1008.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:01 wm-bot2: Finished rebooting node cloudcephosd1007.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Rebooting node cloudcephosd1007.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Finished rebooting node cloudcephosd1006.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Rebooting node cloudcephosd1006.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Finished rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:45 wm-bot2: Rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:45 wm-bot2: Finished rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:20 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:20 wm-bot2: Finished rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:56 wm-bot2: Rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:56 wm-bot2: Finished rebooting node cloudcephosd1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Rebooting node cloudcephosd1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Finished rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:07 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:07 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 16:49 wm-bot2: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:48 wm-bot2: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:46 wm-bot2: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:46 wm-bot2: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 16:39 wm-bot2: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:39 wm-bot2: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:41 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:41 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 15:40 wm-bot2: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:40 wm-bot2: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:37 wm-bot2: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:37 wm-bot2: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:30 wm-bot2: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:30 wm-bot2: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:29 wm-bot2: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:29 wm-bot2: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:25 wm-bot2: Rebooting node cloudcephmon2006-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:25 wm-bot2: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Rebooting node cloudcephmon2005-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:19 wm-bot2: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:19 wm-bot2: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:18 wm-bot2: Finished rebooting the nodes ['cloudcephmon1001', 'cloudcephmon1002', 'cloudcephmon1003'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:18 wm-bot2: Finished rebooting node cloudcephmon1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:15 wm-bot2: Rebooting node cloudcephmon1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:15 wm-bot2: Finished rebooting node cloudcephmon1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Finished rebooting node cloudcephmon1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:08 wm-bot2: Rebooting node cloudcephmon1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:08 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye

=== 2022-12-17 ===
* 07:50 taavi: deleted project packagist-mirror per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge#packagist-mirror

=== 2022-12-16 ===
* 19:36 volans: restarted sshd twice on bastion-restricted-eqiad1-02 to debug SSH connections for [[phab:T319401|T319401]]
* 08:46 dcaro: restart designate-sink on both cloudservice hosts ([[phab:T322279|T322279]])
* 08:45 dcaro: restart designate-sink on both cloudservice hosts

=== 2022-12-07 ===
* 22:07 andrewbogott: systemctl restart libvirt-guests.service  on cloudvirt1019 to get ceph/rbd working on VMS on this hypervisor

=== 2022-12-03 ===
* 19:24 taavi: restart designate-sink on both cloudservices hosts

=== 2022-11-30 ===
* 20:03 andrewbogott: changing all rabbitmq queues to quorum queues.  Will be noisy!  [[phab:T318816|T318816]]
* 02:54 wm-bot2: Upgraded and rebooted host cloudbackup2002.codfw.wmnet - cookbook ran by andrew@bullseye

=== 2022-11-28 ===
* 13:00 wm-bot2: unset cloudvirt1043.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 10:28 wm-bot2: Drained cloudvirt1043.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:19 wm-bot2: Set cloudvirt cloudvirt1043.eqiad.wmnet maintenance (downtime id: bb94dd24-fef9-4c9c-8f79-{{Gerrit|b6e15023ce69}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:18 wm-bot2: Draining cloudvirt1043.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-25 ===
* 10:54 wm-bot2: deleted VM canary2001-dev-2 from cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 10:54 wm-bot2: created VM canary2001-dev-3 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 10:53 wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:5b2ca632-2ea0-4007-9b40-{{Gerrit|4f84f8e2428b}}) - cookbook ran by arturo@nostromo
* 10:46 wm-bot2: Created new flavor: g3.cores1.ram1.disk20.admin (id:fecbd56d-0969-45f3-80fd-{{Gerrit|2b463a5b6270}}) - cookbook ran by arturo@nostromo

=== 2022-11-24 ===
* 16:53 wm-bot2: deleted VM canary2001-dev-1 from cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:53 wm-bot2: created VM canary2001-dev-2 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2003-dev-1 in cloudvirt2003-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2002-dev-1 in cloudvirt2002-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2001-dev-1 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:36 wm-bot2: Created new flavor: cloudvirt-canary-ceph (id:0d06701a-2845-4298-b2b4-{{Gerrit|fabf8b1ddcbb}}) - cookbook ran by arturo@nostromo
* 13:03 wm-bot2: unset cloudvirt1044.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 11:51 wm-bot2: Drained cloudvirt1044.eqiad.wmnet - cookbook ran by arturo@nostromo
* 11:37 wm-bot2: Set cloudvirt cloudvirt1044.eqiad.wmnet maintenance (downtime id: 10076ecc-0f94-4d56-9bbd-{{Gerrit|bebb48bdc126}}, use this to unset) - cookbook ran by arturo@nostromo
* 11:35 wm-bot2: Draining cloudvirt1044.eqiad.wmnet - cookbook ran by arturo@nostromo
* 10:16 dcaro: removed ip6 dns name entry from nb for coluddb* ([[phab:T323550|T323550]])
* 09:53 dcaro: removed ip6 dns entry from nb for coluddb1013 ([[phab:T323550|T323550]])

=== 2022-11-23 ===
* 15:00 wm-bot2: unset cloudvirt1045.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Drained cloudvirt1045.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:35 wm-bot2: Set cloudvirt cloudvirt1045.eqiad.wmnet maintenance (downtime id: 2386b468-0f21-4ecb-91e2-{{Gerrit|e19ace66881d}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:34 wm-bot2: Draining cloudvirt1045.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:20 wm-bot2: unset cloudvirt1046.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 12:17 wm-bot2: Drained cloudvirt1046.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:04 wm-bot2: Set cloudvirt cloudvirt1046.eqiad.wmnet maintenance (downtime id: 6291d38e-c04c-4aa0-88da-{{Gerrit|2b329874a9b9}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:03 wm-bot2: Draining cloudvirt1046.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:02 wm-bot2: unset cloudvirt1047.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 11:32 arturo: [codfw1dev] created project cloudvirt-canary
* 10:13 wm-bot2: Drained cloudvirt1047.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:01 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 2c7ccc17-2be3-427d-aaec-{{Gerrit|57fadca0de5b}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:00 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-22 ===
* 13:26 wm-bot2: unset cloudvirt1048.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 12:15 wm-bot2: unset cloudvirt1049.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 10:58 wm-bot2: Drained cloudvirt1049.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:37 wm-bot2: Set cloudvirt cloudvirt1049.eqiad.wmnet maintenance (downtime id: 3ae0a20d-3cf0-4eba-bea6-{{Gerrit|45aa61d8ad00}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:36 wm-bot2: Draining cloudvirt1049.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:21 wm-bot2: Unset cloudvirt cloudvirt1050.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo

=== 2022-11-21 ===
* 16:19 wm-bot2: Drained cloudvirt1050.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:11 wm-bot2: Set cloudvirt cloudvirt1050.eqiad.wmnet maintenance (downtime id: 0b154a93-a9d3-4ac7-bcfc-{{Gerrit|b67c49abe97b}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:09 wm-bot2: Draining cloudvirt1050.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:07 wm-bot2: Unset cloudvirt cloudvirt1051.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 15:18 wm-bot2: Drained cloudvirt1051.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 15:02 wm-bot2: Set cloudvirt cloudvirt1051.eqiad.wmnet maintenance (downtime id: 1de1174b-ec46-47a3-911c-{{Gerrit|b5808ce37028}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 15:00 wm-bot2: Draining cloudvirt1051.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 14:57 wm-bot2: Unset cloudvirt cloudvirt1052.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 12:53 wm-bot2: Drained cloudvirt1052.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:30 wm-bot2: Set cloudvirt cloudvirt1052.eqiad.wmnet maintenance (downtime id: 30db8a9b-08db-456f-8106-{{Gerrit|53188ff5f989}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:29 wm-bot2: Draining cloudvirt1052.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:15 wm-bot2: Unset cloudvirt cloudvirt1053.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 10:13 arturo: drained cloudvirt1053 in preparation for reimage (was spare anyway)

=== 2022-11-18 ===
* 13:37 arturo: [codfw1dev] reimaged cloudvirt2001-dev and cloudvirt2002-dev
* 11:36 wm-bot2: Set cloudvirt cloudvirt2001-dev.codfw.wmnet maintenance (downtime id: 2fb9df34-fc2d-45b9-b21f-{{Gerrit|c0ec09008b92}}, use this to unset) - cookbook ran by arturo@nostromo
* 11:36 wm-bot2: Draining cloudvirt2001-dev.codfw.wmnet - cookbook ran by arturo@nostromo

=== 2022-11-16 ===
* 20:07 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:56 wm-bot2: Upgraded and rebooted host cloudservices2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:16 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 18:59 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 12:52 arturo: failovered cloudgw1002 into cloudgw1001 for reimage, IRC bots were briefly disconnected

=== 2022-11-14 ===
* 20:22 wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye
* 20:07 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 19:42 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye
* 19:28 andrewbogott: beginning OpenStack upgrade in eqiad1 -- [[phab:T305828|T305828]]
* 19:22 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 19:14 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 19:08 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 12:52 arturo: cleanup old network vlan interface names from /etc/network/interfaces in cloudnet1005/1006

=== 2022-11-11 ===
* 13:24 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: edad3915-b7c6-4b23-bb9c-{{Gerrit|ab13b04a41c5}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:23 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:17 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: a09a9868-8aae-4dc0-8510-{{Gerrit|a3923b703060}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:16 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-10 ===
* 16:19 wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: 346013ec-ce4e-497e-ad65-{{Gerrit|2d215b14998c}}, use this to unset). ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:18 wm-bot2: Draining 'cloudvirt2002-dev.codfw.wmnet'. ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:13 wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: a40a8487-22ee-4bc6-bbe4-{{Gerrit|694a615e3bf5}}, use this to unset). ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-08 ===
* 11:17 taavi: backfilling security groups for metricsinfra access on all projects [[phab:T288108|T288108]]

=== 2022-11-07 ===
* 21:01 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 20:40 andrewbogott: upgrading eqiad1 designate to version 'yoga'

=== 2022-11-04 ===
* 17:57 andrewbogott: removing cinderv2 API endpoints from keystone catalog; this is deprecated and removed in Yoga.  prep for [[phab:T305828|T305828]]

=== 2022-11-03 ===
* 19:24 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 00:08 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye

=== 2022-11-02 ===
* 23:39 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:38 wm-bot2: Upgraded and rebooted host cloudnet2006-dev.codfw.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Upgraded and rebooted host cloudnet2005-dev.codfw.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:13 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:00 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 22:54 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 20:29 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye

=== 2022-10-31 ===
* 13:09 arturo: restart keepalived on all 4 cloudgw servers to run them with `-D` in /etc/default/keepalived to further debug [[phab:T320975|T320975]]

=== 2022-10-26 ===
* 16:18 wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:bf48880d-0c1b-4c2a-8e8b-{{Gerrit|778d28b16561}}) ([[phab:T319446|T319446]]) - cookbook ran by dcaro@vulcanus
* 09:34 taavi: running wmcs-puppetcertleaks in delete mode
* 09:09 taavi: running wmcs-novastats-dnsleaks in delete mode

=== 2022-10-25 ===
* 16:03 arturo: [codfw1dev] [[phab:T321220|T321220]] root@cloudcontrol2001-dev:~# openstack subnet create magnum --no-dhcp --network 57017d7c-3817-429a-8aa3-{{Gerrit|b028de82cdcc}} --ip-version 4 --gateway auto --subnet-range 192.168.0.0/24
* 14:38 arturo: [codfw1dev] restart neutron-l3-agent in cloudnet2005-dev, it was dead after rabbit connectivity problems

=== 2022-10-24 ===
* 18:50 wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Finished rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:47 wm-bot2: Rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:47 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by andrew@bullseye
* 18:38 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by andrew@bullseye
* 18:21 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:21 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl

=== 2022-10-20 ===
* 23:23 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:23 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:20 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:20 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:19 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:16 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:07 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:07 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:06 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:06 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 101c41d1-d65d-4088-b6d2-{{Gerrit|eac859e45ef8}}, use this to unset). - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:01 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:57 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 7197ce34-cc57-4677-a1a9-{{Gerrit|05e20cd0dd80}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:38 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 68e4cc1d-9def-444e-84a7-{{Gerrit|21b0b5adfa72}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 73388c1e-369b-40af-a2c1-{{Gerrit|96936128c324}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 12feeca2-ea59-48e9-9235-{{Gerrit|1cecf5b384cd}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 1b34587f-2770-4d2e-bb31-{{Gerrit|c8ad14632d39}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:24 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:13 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: e75b00eb-7f58-4821-8139-{{Gerrit|3dfc6e97a92a}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 9151511e-bcc0-4367-a976-{{Gerrit|7cff060308aa}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 241c0bd9-c3cf-40e4-95dc-{{Gerrit|9b51d9823fe9}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 7a53c274-1d5f-4c94-9a0d-{{Gerrit|721c3e8f7239}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:10 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:10 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:06 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:54 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:54 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:51 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 0b75b44c-1efe-4edf-8f5e-{{Gerrit|42a67e8d3b13}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: aeb9c3a6-961a-4873-85d4-{{Gerrit|929248aebb8b}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: eb14cc04-3293-4cf8-a46f-{{Gerrit|1f87d8b0bcc4}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 84a3cd99-2643-495a-86b6-{{Gerrit|7ae80b11a30d}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:32 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:32 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:28 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:28 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 01a0b69c-2e27-4331-9635-{{Gerrit|403a668dac29}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:27 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:27 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:23 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: ea29f65f-6d86-4568-8db9-{{Gerrit|a85faa827447}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:22 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:22 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:18 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:18 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:16 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:14 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: d6b6bb4d-c7c0-4bd2-9a02-{{Gerrit|dee9a5ec6a3e}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 4e109149-7d67-403f-8b21-{{Gerrit|f829235ea491}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: 341f4b8c-6d6e-4190-9f2d-{{Gerrit|a1a1483abadd}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: f76e8f14-920e-4d51-a44e-{{Gerrit|321a55dcb0c5}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:56 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:56 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:54 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:35 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:34 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 0b0d8090-4d31-4d43-8545-{{Gerrit|c25209e2ef58}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:31 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 3d928554-a79c-419a-bcc4-{{Gerrit|c8d63791d8e7}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 3d3c8aa5-45f7-429e-a9d7-{{Gerrit|b5181b29dc13}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: b9027020-8a90-4c40-b0e6-{{Gerrit|6c8767f87917}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: f0d60fab-64e9-4da2-826c-{{Gerrit|229d31d0fbc2}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:16 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:14 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:11 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: b65da7e2-9fd2-4a1e-8dd4-{{Gerrit|88ca65936ae4}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: cbbf114c-9a4c-4dd2-9b58-{{Gerrit|50da8bd896ca}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 5969beaf-72bf-4af9-ae4d-{{Gerrit|8e5c3331e78e}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: b8f7389d-79b4-411d-b3d5-{{Gerrit|ec8f92eba101}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:53 wm-bot2: Safe reboot of 'cloudvirt1051.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:53 wm-bot2: Unset cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:51 wm-bot2: Safe reboot of 'cloudvirt1049.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:51 wm-bot2: Unset cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:50 wm-bot2: Safe reboot of 'cloudvirt1048.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:50 wm-bot2: Unset cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Drained 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: c5dbfa7b-72fc-4156-8257-{{Gerrit|af224a725b78}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: c1a3e92c-cb04-46e4-ac5d-{{Gerrit|784c79601b05}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance (downtime id: 62fdc4ee-c8d5-4892-aeb9-{{Gerrit|a681cdcbc84b}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 9ec407cd-5c00-407a-a57d-{{Gerrit|794f6c68f947}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Set cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance (downtime id: 712683ea-d58f-484b-8efb-{{Gerrit|89d1c21aa0e3}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Safe rebooting 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Draining 'cloudvirt1049.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Safe rebooting 'cloudvirt1049.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Draining 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Safe rebooting 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:25 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:25 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:21 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:04 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 18fae8d8-7353-4f67-90d7-{{Gerrit|8df9b3fb1ccb}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:03 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:03 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:01 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 3d3cffa3-abc6-4901-83d9-{{Gerrit|3bcc4b02fd3c}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:00 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:00 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:54 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:54 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:46 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:46 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye

=== 2022-10-15 ===
* 17:38 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki Slevinski # [[phab:T320867|T320867]]

=== 2022-10-13 ===
* 12:19 wm-bot2: OSDs (['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034'] ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 11:30 wm-bot2: OSDs (['cloudcephosd1025', 'cloudcephosd1026']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus

=== 2022-10-10 ===
* 14:01 dcaro: test2
* 01:22 andrewbogott: restarting designate-sink on cloudservices100[45], possible example of [[phab:T316614|T316614]]

=== 2022-10-09 ===
* 12:04 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki DatGuy # [[phab:T320301|T320301]]

=== 2022-10-07 ===
* 13:40 andrewbogott: dhinus is resetting rabbitmq cluster in an attempt to resolve a suspected (by Andrew) split-brain
* 11:33 arturo: rabbitmq-server.service @ cloudrabbit1002 is again up and running ([[phab:T320232|T320232]])
* 10:24 arturo: stopping rabbitmq-server.service @ cloudrabbit1002 ([[phab:T320232|T320232]])
* 10:19 arturo: restarting nova-conductor in all 3 cloudcontrols ([[phab:T320232|T320232]])
* 09:45 arturo: restarting rabbitmq-server.service @ cloudrabbit1002 ([[phab:T320232|T320232]])

=== 2022-10-06 ===
* 15:55 arturo: cloudnet1005 &amp; cloudnet1006 now in service. Secom cloudnet1003 &amp; cloudnet1004. Drop neutron agents, etc. ([[phab:T316284|T316284]])
* 11:54 arturo: rebooting cloudnet1005/1006 to see if they have the right network config ([[phab:T316284|T316284]])
* 11:50 arturo: set neutron l3 agents on cloudnet1005/1006 as down `root@cloudcontrol1005:~# neutron agent-update --admin-state-down &lt;uuid&gt;` ([[phab:T316284|T316284]])
* 11:40 arturo: [codfw1dev] rebooting both network nodes to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/839492
* 10:14 arturo: [codfw1dev] restart neutron-l3-agent on cloudnet2006-dev, it was dead

=== 2022-10-05 ===
* 14:40 wm-bot2: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T319418|T319418]]) - cookbook ran by fran@wmf3169
* 14:40 wm-bot2: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T319418|T319418]]) - cookbook ran by fran@wmf3169
* 14:28 arturo: adding cloudinstances2b-gw router to l3 agents on cloudnet1005/1006 ([[phab:T316284|T316284]])
* 13:11 wm-bot2: Added 1 new OSDs ['cloudcephosd1034.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 13:11 wm-bot2: Added OSD cloudcephosd1034.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 13:02 wm-bot2: Finished rebooting node cloudcephosd1034.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Rebooting node cloudcephosd1034.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Adding OSD cloudcephosd1034.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Adding new OSDs ['cloudcephosd1034.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169

=== 2022-10-04 ===
* 16:40 wm-bot2: Added 1 new OSDs ['cloudcephosd1033.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 16:40 wm-bot2: Added OSD cloudcephosd1033.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:20 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 14:17 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:59 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:56 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:55 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:55 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169

=== 2022-09-30 ===
* 14:52 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169
* 14:52 wm-bot2: Added OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:48 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:48 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 64eac5c6-4b1d-4269-98fd-{{Gerrit|8e5bed42ce40}}, use this to unset). ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:09 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:05 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: fb99c967-b974-4314-a2fa-{{Gerrit|31ed0e883dd3}}, use this to unset). - cookbook ran by andrew@buster
* 14:04 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 13:30 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169
* 13:30 wm-bot2: Added OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:26 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:26 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 13:16 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:16 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 13:08 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:08 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 12:45 wm-bot2: Finished rebooting node cloudcephosd1031.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:42 wm-bot2: Rebooting node cloudcephosd1031.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:41 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:41 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:26 arturo: [codfw1dev] cloudnet2005/2006-dev are now on a single NIC setup ([[phab:T318824|T318824]])
* 11:44 arturo: sysctl change (cleanup) on cloudnet1003/1004

=== 2022-09-27 ===
* 10:48 wm-bot2: Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:35 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:26 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:23 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:22 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:22 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:17 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:09 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169

=== 2022-09-26 ===
* 18:37 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 18:37 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:33 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:31 andrewbogott: rebooting cloudvirt1028 for [[phab:T317391|T317391]]
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance (downtime id: 6d1c4c53-76b9-493f-8c44-{{Gerrit|eb0413dfb9d0}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance (downtime id: 6bb80b65-616c-4e45-be4f-{{Gerrit|d2cdd8abb2bf}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance (downtime id: a3bba7e7-bcd3-482d-a486-{{Gerrit|06b6f53fd899}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:18 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8fb59651-fd42-4aee-a978-{{Gerrit|36bc7f79b4d5}}, use this to unset). - cookbook ran by andrew@buster
* 18:18 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:17 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:01 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:01 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:58 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:51 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:51 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:47 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:47 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:47 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:45 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:38 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: abfc35bb-331e-4d7e-bb92-{{Gerrit|35e675abce32}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:34 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:33 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: d4fe6ea8-76eb-45f7-b6cf-{{Gerrit|66f54ba9801c}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:33 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: c0671bda-9aae-4960-85be-{{Gerrit|58aaa9c8e021}}, use this to unset). - cookbook ran by andrew@buster
* 17:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:32 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:32 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:28 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:28 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:24 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: be6559f9-4d72-4492-b9b9-{{Gerrit|23c636d6554e}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:23 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:23 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:21 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 237dfb1b-5382-408d-aa35-{{Gerrit|1aa1cfe4c5af}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 088b97cd-0cf2-4c27-a67f-{{Gerrit|077ffa1c1c5e}}, use this to unset). - cookbook ran by andrew@buster
* 17:16 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:16 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 91625000-aa6d-4887-bf34-{{Gerrit|ef7785a4af4a}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:16 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:15 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:15 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:15 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:15 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 7364c4fe-9f6b-4539-b6fa-{{Gerrit|1e768b602a1b}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:14 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:14 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:12 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:12 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: cb8f8a21-73c0-4f54-9412-{{Gerrit|074d82582cb4}}, use this to unset). - cookbook ran by andrew@buster
* 17:11 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:11 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:09 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: d23dabd2-3bfc-4ce0-9533-{{Gerrit|cd1cf910f8e1}}, use this to unset). - cookbook ran by andrew@buster
* 17:08 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:08 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:05 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 4d7ddae1-f621-4dd5-8616-{{Gerrit|29b7699b692f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 4628dd6f-5b49-417f-b6ab-{{Gerrit|5c41992192b4}}, use this to unset). - cookbook ran by andrew@buster
* 17:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: c95529cf-b2a5-4553-bd6e-{{Gerrit|6ad4dcb2a105}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:01 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 36c68112-a964-4a7c-a093-{{Gerrit|8a6d481dea7d}}, use this to unset). - cookbook ran by andrew@buster
* 17:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:00 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:00 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:49 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:49 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:45 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:31 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 1244c159-65bb-476e-a702-{{Gerrit|8f43c253e499}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:30 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:30 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:22 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 13:19 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 13:18 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:18 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 12:32 dcaro: Changed the collation of labsdbaccount db to utf8mb4_bin ([[phab:T318047|T318047]])
* 10:14 arturo: deployed new version of maintain-dbusers ([[phab:T318047|T318047]])

=== 2022-09-25 ===
* 15:06 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:06 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: a050e47f-3a63-41ff-accb-{{Gerrit|3993c1e8b593}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:05 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:05 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:58 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:58 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 1a49d773-4dc9-4a6f-bd49-{{Gerrit|8663db77ce66}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:57 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:57 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:54 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: ab52dd42-68a6-4159-b345-{{Gerrit|e5625d209b57}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:53 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:53 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 10337415-3095-4834-8538-{{Gerrit|b3a64bd6b18d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:53 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:53 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 4cdbed3a-3775-4913-8c3b-{{Gerrit|afd57ae1ca55}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-24 ===
* 17:37 andrewbogott: restarting neutron api on cloudcontrol1006; cause of outage unknown
* 17:35 andrewbogott: restarting neutron-linuxbridge-agent on cloudvirt1022

=== 2022-09-22 ===
* 15:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:10 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04849437-a304-45a1-a037-{{Gerrit|538741a2f801}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:09 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:09 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:07 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:07 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:04 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 8810a919-61e7-4ba5-af7f-{{Gerrit|c41d1e1c8c8b}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:59 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:59 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:56 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: b6f8e0b6-f35c-41fd-8035-{{Gerrit|6efde61db3a3}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 784bff2c-8160-44dc-9bcf-{{Gerrit|ff23055a0527}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:40 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:40 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:39 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:38 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:38 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 21152e9e-bc67-4024-b68e-{{Gerrit|fd671c398642}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: f07ddb1d-e8da-43f3-8140-{{Gerrit|bc66789d37c8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:36 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:36 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:33 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:33 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:30 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 4e4b03fc-e3f1-440e-9097-{{Gerrit|cce5edaa1f08}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: d4bb2160-f492-4ddc-a5b7-{{Gerrit|eeee37007d14}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9bf00a6a-3697-4201-9a6e-{{Gerrit|76d499eccfa1}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:13 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:13 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:47 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f2888490-1804-4b23-b7b0-{{Gerrit|677853fb9ef7}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8a3e8b0a-ced3-4667-a121-{{Gerrit|66df673c7ed8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:43 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 35c761b3-f0de-4b23-9b11-{{Gerrit|2ed2e474c89f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 6121a0e1-754a-47fa-b51b-{{Gerrit|265f6386f396}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:40 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:40 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:39 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 189ae1bb-c69d-4eb0-aee9-{{Gerrit|90ab1f492aa8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:38 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:38 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:36 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: acc53776-68bf-47a4-bb82-{{Gerrit|c571b3df2945}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:31 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 07770ae7-3e7e-4615-8174-{{Gerrit|513767f95b1d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:30 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:30 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 7ae20a7d-8bff-4bc2-91c5-{{Gerrit|b0005c8164d1}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: b596f855-8019-49d2-9657-{{Gerrit|0d513ad7acb5}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:25 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 9172c785-d2d6-46e5-bed3-{{Gerrit|2f49d1033f78}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:23 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:23 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:19 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 85b4e0e2-e635-485f-9ce5-{{Gerrit|341981b3887f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 9494c283-02c8-42da-98c3-{{Gerrit|139b4d119ef8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: d181f301-249f-4b5a-bd85-{{Gerrit|47d87262d0ed}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-20 ===
* 21:02 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 21:02 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:58 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:58 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 1c4246a4-9cee-4423-8cd1-{{Gerrit|4f52f61d503f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:53 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:50 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:50 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:49 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 82665ecc-4431-48fe-b255-{{Gerrit|7e9d518be217}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:48 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:48 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:47 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: cd284562-e3b0-4504-9977-{{Gerrit|2b18032bbf3c}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:45 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 8fb2a646-16a5-4183-94a1-{{Gerrit|ee6bbbf029fa}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:44 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:44 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: d561fe45-1582-4cd8-bbc9-{{Gerrit|a356c39f3330}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 351c365c-0228-4907-a279-{{Gerrit|01795b1e62ce}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 3beec1dd-0132-4f5c-adce-{{Gerrit|5a7dc56060b6}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:30 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:29 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:28 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:28 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:26 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:24 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:20 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:16 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 353a8527-ad5e-4898-937c-{{Gerrit|303d16801e28}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: fc235146-f070-4723-9503-{{Gerrit|e20cbb877755}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: cdf712ac-c083-4a63-af83-{{Gerrit|514ca5cdc76d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:05 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:01 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:01 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 8f2d2d32-e99e-4e2c-9472-{{Gerrit|ca33b577d01f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 66ee7d87-cee1-4c8f-b5c3-{{Gerrit|5f58a6cea336}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:59 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:59 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:58 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:58 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:58 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 69035b38-9910-46f1-9582-{{Gerrit|3105c1ba4d16}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:57 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:57 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:52 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 5a94c9d1-c956-4b03-a92b-{{Gerrit|b112810906a4}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:51 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:51 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:50 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 216a9120-4b9a-4be3-99db-{{Gerrit|3fd5fdc25146}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:49 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:49 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 46fb1964-3cf4-47fd-ad79-{{Gerrit|475f33c8ed38}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: b04a4f8c-db82-4258-8608-{{Gerrit|2a78ec9193ed}}, use this to unset). - cookbook ran by andrew@buster
* 19:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 7ae7c258-ecb8-47d1-a991-{{Gerrit|3fa617e305bf}}, use this to unset). - cookbook ran by andrew@buster
* 19:45 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 377fb0e5-132e-4e7a-9079-{{Gerrit|37937c3d42bf}}, use this to unset). - cookbook ran by andrew@buster
* 19:44 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 6c4222ae-796e-4890-8820-{{Gerrit|9e7ce5438f2a}}, use this to unset). - cookbook ran by andrew@buster
* 19:43 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:43 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:38 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:34 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:34 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:31 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: eff98e2e-ae99-41d0-a68e-{{Gerrit|671d941fcf21}}, use this to unset). - cookbook ran by andrew@buster
* 19:30 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5310f1eb-0425-4f59-8fb5-{{Gerrit|1d9c6f6d2f23}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:29 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:25 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: d8ab4682-026a-4ddc-bfc5-{{Gerrit|48166bd9789a}}, use this to unset). - cookbook ran by andrew@buster
* 19:24 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:23 wm-bot2: Safe reboot of 'cloudvirt1048.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:23 wm-bot2: Unset cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: eb2f9d94-8ea5-48d5-a336-{{Gerrit|97dd407b1ce3}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Safe rebooting 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:18 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:14 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 305fe2d7-913e-4638-95c2-{{Gerrit|4e810dc6b2a3}}, use this to unset). - cookbook ran by andrew@buster
* 19:14 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5bb3a66b-9450-4230-8d09-{{Gerrit|d78473d72ba2}}, use this to unset). - cookbook ran by andrew@buster
* 19:13 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:13 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:07 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: b1e39b49-3a88-4e42-913c-{{Gerrit|9892eafad447}}, use this to unset). - cookbook ran by andrew@buster
* 19:06 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:05 andrewbogott: putting cloudvirt1049-1052 into 'ceph' pool, taking out of 'spare' pool. cloudvirt1053 will remain our only spare.
* 19:02 wm-bot2: Safe reboot of 'cloudvirt1049.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:02 wm-bot2: Unset cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:00 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:00 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:59 wm-bot2: Safe reboot of 'cloudvirt1051.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:59 wm-bot2: Unset cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Drained 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Set cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance (downtime id: 8f24ea75-e107-4a13-8bd6-{{Gerrit|950b941b8e03}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Draining 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Safe rebooting 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:57 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:57 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:56 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:56 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 5c3e8fa4-cd1e-43f4-a423-{{Gerrit|ba7fe2459d4e}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Drained 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Set cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance (downtime id: e3fe22d6-932c-4502-8838-{{Gerrit|82cdc4c8f1c6}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:54 wm-bot2: Draining 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:54 wm-bot2: Safe rebooting 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:53 wm-bot2: Safe reboot of 'cloudvirt1053.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:53 wm-bot2: Unset cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: fd824b92-b6ff-4631-a2d4-{{Gerrit|debb25d48223}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:51 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:49 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 52d5cd5b-0f56-453a-85f8-{{Gerrit|0b5e656ae7f9}}, use this to unset). - cookbook ran by andrew@buster
* 18:49 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:49 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: b92f66a3-a487-4515-b68b-{{Gerrit|1c257a57b893}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:48 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-19 ===
* 20:07 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 4bfca6cd-dfbc-4a79-9203-{{Gerrit|b432bfeee5c2}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:41 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:21 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 70479325-609b-4349-9094-{{Gerrit|739eb9b3bb30}}, use this to unset). - cookbook ran by andrew@buster
* 19:21 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-09-14 ===
* 16:57 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
* 11:01 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:58 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:57 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:57 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:09 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:09 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:06 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:06 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 09:46 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-09-13 ===
* 12:16 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:16 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:11 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:11 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:36 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-09-10 ===
* 15:37 andrewbogott: restarting nova-conductor service (and possibly others, in response to lots of unanswered rabbitmq messages)

=== 2022-09-08 ===
* 19:12 andrewbogott: restarting nginx on proxy-03.project-proxy.eqiad1.wikimedia.cloud

=== 2022-09-07 ===
* 10:18 wm-bot2: Added OSD cloudcephosd1032.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:14 wm-bot2: Finished rebooting node cloudcephosd1032.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Rebooting node cloudcephosd1032.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Adding OSD cloudcephosd1032.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Adding new OSDs ['cloudcephosd1032.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:41 dhinus: Temporarily removing cloudcephosd1030 from Ceph cluster (https://phabricator.wikimedia.org/T314870)

=== 2022-08-30 ===
* 14:59 andrewbogott: manually marking most eqiad1 cloud* servers down in icinga for [[phab:T296561|T296561]]
* 10:43 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:39 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:38 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:38 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:39 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:14 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:11 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-25 ===
* 15:14 wm-bot2: Added 1 new OSDs ['cloudcephosd1029.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 15:14 wm-bot2: Added OSD cloudcephosd1029.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 15:02 wm-bot2: Finished rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:59 wm-bot2: Rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:58 wm-bot2: Adding OSD cloudcephosd1029.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:58 wm-bot2: Adding new OSDs ['cloudcephosd1029.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-24 ===
* 22:07 andrewbogott: replaced cloudservices1003 with cloudservices1005 [[phab:T304888|T304888]]
* 10:45 wm-bot2: Added 1 new OSDs ['cloudcephosd1028.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:45 wm-bot2: Added OSD cloudcephosd1028.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Finished rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:34 wm-bot2: Rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:33 wm-bot2: Adding OSD cloudcephosd1028.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:33 wm-bot2: Adding new OSDs ['cloudcephosd1028.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-23 ===
* 13:46 wm-bot2: Added 1 new OSDs ['cloudcephosd1027.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:46 wm-bot2: Added OSD cloudcephosd1027.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:27 wm-bot2: Finished rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:24 wm-bot2: Rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:22 wm-bot2: Adding OSD cloudcephosd1027.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:22 wm-bot2: Adding new OSDs ['cloudcephosd1027.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-21 ===
* 21:12 andrewbogott: restarted neutron-dhcp-agent on cloudnet1003. it was claiming to be unable to contact Rabbit but seems happy after a restart

=== 2022-08-20 ===
* 07:39 dcaro_away: cloudvirt1023 is back up, VMs are starting to recover ([[phab:T315718|T315718]])
* 07:23 dcaro_away: cloudvirt1023 seems to have gotten some hardware issue from racadm lclog view &quot;System CPU Resetting.&quot;, rebooting and doing memory checks ([[phab:T315718|T315718]])

=== 2022-08-19 ===
* 17:06 taavi: [codfw1dev] restart mariadb on clouddb2002-dev to pick up certificate config changes [[phab:T310795|T310795]]

=== 2022-08-18 ===
* 13:25 wm-bot2: Added 1 new OSDs ['cloudcephosd1026.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:25 wm-bot2: Added OSD cloudcephosd1026.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:15 wm-bot2: Finished rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:12 wm-bot2: Rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:10 wm-bot2: Adding OSD cloudcephosd1026.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:10 wm-bot2: Adding new OSDs ['cloudcephosd1026.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 07:29 dcaro: Starting up all the osd daemons on cloudcephosd1025 ([[phab:T314870|T314870]])

=== 2022-08-17 ===
* 10:50 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:49 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:50 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:49 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:49 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:16 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:16 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:16 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-16 ===
* 22:39 andrewbogott: replacing the now-rebuilt cloudvirt1025 in 'ceph' aggregate and removing it from the 'maintenance' aggregate
* 17:41 andrewbogott: removing cloudvirt1025 from the 'ceph' aggregate and adding it to the 'maintenance' aggregate
* 17:40 andrewbogott: reimaging cloudvirt1025 after I accidentally deleted the hw raid
* 17:38 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1003@rbd#RBD --newhost cloudcontrol1005@rbd#RBD
* 17:37 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1004@rbd#RBD --newhost cloudcontrol1006@rbd#RBD
* 16:26 wm-bot2: Ceph cluster at eqiad1 set out of maintenance. - cookbook ran by dcaro@vulcanus
* 15:43 wm-bot2: Restarting the osd daemons from nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,c
* 15:42 wm-bot2: Finished restarting all the OSD daemons from the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
* 15:38 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:02 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 12:59 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus

=== 2022-08-14 ===
* 18:36 taavi: deleted the http keystone endpoints from the keystone service catalog

=== 2022-08-11 ===
* 13:57 andrewbogott: decommissioning cloudcontrol1003 + cloudcontrl1004. I backed up $home in case anyone needs their files.
* 08:42 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:42 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:42 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-10 ===
* 13:10 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-04 ===
* 17:16 taavi: deleted all scheduler_fanout_ rabbit queues in an attempt to fix scheduling
* 16:32 taavi: restart neutron-l3-agent to pick up rabbit config changes
* 15:12 andrewbogott: stopping rabbitmq on cloudcontrol1xxx
* 09:57 taavi: stop wikitech_run_jobs.timer on labweb1001/1002, hosts pending decom

=== 2022-08-03 ===
* 20:55 andrewbogott: root@tools-checker-04:~# systemctl restart uwsgi-toolschecker_cron.service
* 20:41 andrewbogott: restarting  neutron-l3-agent.service on cloudnet1003 and 1004. The agent was routing properly but had lost touch with rabbitmq

=== 2022-08-02 ===
* 14:07 andrewbogott: shutting down codfw1dev ceph cluster according to https://docs.mirantis.com/mcp/q4-18/mcp-operations-guide/scheduled-maintenance-power-outage/power-off-ceph-cluster.html
* 13:54 andrewbogott: shutting down basically all of codfw1dev to support pdu maintenance -- all the ceph OSDs will lose power so best to have everything stopped.

=== 2022-07-27 ===
* 19:32 andrewbogott: switching the openstack.eqiad1.wikimedia.cloud endpoint from cloudcontrol1004 to 1006, https://gerrit.wikimedia.org/r/c/operations/dns/+/817878/2/templates/wikimediacloud.org#54
* 16:33 andrewbogott: here is a test message in the admin channel

=== 2022-07-25 ===
* 13:43 andrewbogott: pooling cloudweb100[34] and depooling labweb100[12] for testing in prep for decomming labweb100[12]

=== 2022-07-22 ===
* 16:41 taavi: depool cloudweb1003/1004 since horizon seems to be having issues
* 16:22 taavi: pooling cloudweb1003/1004 now that grant issues are sorted

=== 2022-07-21 ===
* 18:26 andrewbogott: depooling cloudweb1003 and 1004 for wikitech, horizon, striker -- pending db grant changes
* 18:06 andrewbogott: pooling cloudweb1003 and 1004 for wikitech, horizon, striker

=== 2022-07-20 ===
* 18:02 dcaro: things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 ([[phab:T313400|T313400]])
* 17:45 bd808: `sudo service striker restart` on labweb1002
* 17:43 bd808: `sudo service striker restart` on labweb1001
* 17:10 dcaro: things seem stable, trying to bring up a fourth rabbit node, cloudcontrol1006 ([[phab:T313400|T313400]])
* 16:26 dcaro: things seem stable, trying to bring up a third, cloudcontrol1005 ([[phab:T313400|T313400]])
* 15:51 dcaro: things seem stable now with one rabbit node, trying to bring up a second ([[phab:T313400|T313400]])
* 14:16 dcaro: stopping rabbin on cloudcontrol1004, leaving only 1003 alive ([[phab:T313400|T313400]])
* 13:17 dcaro: restarting the whole rabbit cluster ([[phab:T313400|T313400]])

=== 2022-07-19 ===
* 16:30 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:30 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 16:26 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:18 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:18 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 16:14 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:01 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:01 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:57 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:57 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 3da2d4f6-5c5b-4a21-9a0b-{{Gerrit|2b010960ed6a}}, use this to unset). - cookbook ran by andrew@buster
* 15:56 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:49 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 8a10505a-107d-4e78-ba84-{{Gerrit|363a7dea8d69}}, use this to unset). - cookbook ran by andrew@buster
* 15:47 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 752aa58b-44d4-4340-b05d-{{Gerrit|911b14f2314d}}, use this to unset). - cookbook ran by andrew@buster
* 15:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: f89f0851-ce3f-4a92-899e-{{Gerrit|0d4638acc9c3}}, use this to unset). - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:45 andrewbogott: adding new hosts to the 'ceph' aggregate: cloudvirt1046, 1047, 1048
* 15:44 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:44 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:42 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:42 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:40 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:38 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:37 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:37 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:33 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:32 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 32d9fdb7-6c42-4cf4-9950-{{Gerrit|6e2442255161}}, use this to unset). - cookbook ran by andrew@buster
* 15:31 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:31 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:16 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 56ce1388-792c-442c-bb89-{{Gerrit|e4c3869b0acf}}, use this to unset). - cookbook ran by andrew@buster
* 15:15 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:15 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:14 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 3f84bf2e-24de-4ebe-8117-{{Gerrit|0f4a5de29d09}}, use this to unset). - cookbook ran by andrew@buster
* 15:14 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:14 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:13 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:13 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:12 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 5418e923-997c-49ae-b937-{{Gerrit|35fc0c3038eb}}, use this to unset). - cookbook ran by andrew@buster
* 15:12 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: c3dfdc2b-0725-4d01-87f2-{{Gerrit|440a5ac4694c}}, use this to unset). - cookbook ran by andrew@buster
* 15:11 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:09 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:04 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:04 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:00 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:54 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:54 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:50 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:46 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 49dbf7de-c58a-4b68-bc7b-{{Gerrit|f001a047f4c7}}, use this to unset). - cookbook ran by andrew@buster
* 14:46 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:46 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:44 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: a4e95168-39b5-452d-8fce-{{Gerrit|7875ae44a62b}}, use this to unset). - cookbook ran by andrew@buster
* 14:44 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:44 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:43 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: b46234ee-9900-4708-a815-{{Gerrit|4324379be48c}}, use this to unset). - cookbook ran by andrew@buster
* 14:43 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:43 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:42 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:39 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:39 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:38 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:38 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 12bc87a9-7602-4795-818c-{{Gerrit|f7151b1c9ead}}, use this to unset). - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:35 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: ba787a26-62db-4662-ade2-{{Gerrit|2e3061b7390d}}, use this to unset). - cookbook ran by andrew@buster
* 14:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:29 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 6f0e79c6-2d91-478e-92df-{{Gerrit|0f65de48212f}}, use this to unset). - cookbook ran by andrew@buster
* 14:29 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 7f577c0f-0182-498f-b198-{{Gerrit|307d51df8c3a}}, use this to unset). - cookbook ran by andrew@buster
* 14:28 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: bd1bff92-bfb8-483d-a719-{{Gerrit|13fbdf3d8b21}}, use this to unset). - cookbook ran by andrew@buster
* 14:27 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:27 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: c9cd7dbb-c5ad-4364-a8e0-{{Gerrit|afb52ef98683}}, use this to unset). - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 4deca8a4-f73f-4b42-b4be-{{Gerrit|acfe61421ed0}}, use this to unset). - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 6d0d58b2-8450-480d-94f2-{{Gerrit|93a257f25575}}, use this to unset). - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:13 dcaro: deleting all the leftover fullstack images (was due to max number of mysql connections reached)
* 04:51 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:47 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:46 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 432747a2-9a15-44d4-ba8a-{{Gerrit|4e7e87eb8b76}}, use this to unset). - cookbook ran by andrew@buster
* 04:45 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1c5adb8c-efcf-4036-bf22-{{Gerrit|0eef364ae399}}, use this to unset). - cookbook ran by andrew@buster
* 04:44 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:44 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:41 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 26feb897-9eb0-40b6-9525-{{Gerrit|4cf44f81638d}}, use this to unset). - cookbook ran by andrew@buster
* 04:41 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:40 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:40 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:39 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: cd2424d3-c9c9-4837-93b1-{{Gerrit|a6d8271e7873}}, use this to unset). - cookbook ran by andrew@buster
* 04:39 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:39 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1e85ec01-81c4-4664-b14f-{{Gerrit|b1cf6417988c}}, use this to unset). - cookbook ran by andrew@buster
* 04:38 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f498aaa4-fad3-42f8-92f4-{{Gerrit|48a98c108456}}, use this to unset). - cookbook ran by andrew@buster
* 04:37 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 0ed281f3-d9b2-4c0a-bf88-{{Gerrit|db42fe848b2f}}, use this to unset). - cookbook ran by andrew@buster
* 04:35 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f417dffc-fa1b-4e65-938a-{{Gerrit|49e54b6fac6d}}, use this to unset). - cookbook ran by andrew@buster
* 04:34 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:33 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 923926e0-7194-4e13-98c4-{{Gerrit|a4f703b2907b}}, use this to unset). - cookbook ran by andrew@buster
* 04:32 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:32 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:29 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: ca8f729b-92ad-45ec-b5da-{{Gerrit|55987b0fc9c2}}, use this to unset). - cookbook ran by andrew@buster
* 04:29 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:29 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: abe6992d-dfba-4e50-993d-{{Gerrit|de0a551a177a}}, use this to unset). - cookbook ran by andrew@buster
* 04:24 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:24 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: df91107d-bf34-49db-9756-{{Gerrit|44ad06162683}}, use this to unset). - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f45ec0b1-83c9-49e0-a3a8-{{Gerrit|897d5a48414f}}, use this to unset). - cookbook ran by andrew@buster
* 04:23 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:18 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 504ae503-bfad-41bd-9079-{{Gerrit|fb53a9c82a62}}, use this to unset). - cookbook ran by andrew@buster
* 04:17 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:17 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 8410faf9-667d-443d-bd17-{{Gerrit|bb3ca8e7f725}}, use this to unset). - cookbook ran by andrew@buster
* 04:15 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:11 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 40441a37-0dd7-44a8-960b-{{Gerrit|f74f98f53618}}, use this to unset). - cookbook ran by andrew@buster
* 04:10 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:10 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:08 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 517cff2b-a643-4714-a31f-{{Gerrit|6bbe0767656a}}, use this to unset). - cookbook ran by andrew@buster
* 04:08 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 50b63ad6-4d8b-4fcf-9c17-{{Gerrit|27a106b6ec52}}, use this to unset). - cookbook ran by andrew@buster
* 04:07 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: d912bdf5-54c7-490a-bdbe-{{Gerrit|ba9a23f07f4b}}, use this to unset). - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: e75f16d0-e42f-4d72-a3bd-{{Gerrit|be0cbc5f200a}}, use this to unset). - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 87f6c417-3703-4f73-9876-{{Gerrit|20f6767dc8e1}}, use this to unset). - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: be14f869-fdb0-4c38-84ff-{{Gerrit|3d1e00d227eb}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: feb93ea7-2c64-4831-b140-{{Gerrit|dad8311dbcef}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: c65a23e3-a809-4a75-88e4-{{Gerrit|29a8b351ecdb}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:00 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:00 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:57 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:56 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8c8f64ca-a4e8-47cf-a2ce-{{Gerrit|23059109fb6a}}, use this to unset). - cookbook ran by andrew@buster
* 03:55 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:55 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:54 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 9d0457e8-a861-46ce-ac35-{{Gerrit|53975a46018e}}, use this to unset). - cookbook ran by andrew@buster
* 03:53 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:53 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:40 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:38 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:30 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 851d3b34-68f7-4c57-920b-{{Gerrit|2a64a9ea573d}}, use this to unset). - cookbook ran by andrew@buster
* 03:29 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:29 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:17 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 1060ab82-d53a-4d48-8e15-{{Gerrit|3198cabcfc2f}}, use this to unset). - cookbook ran by andrew@buster
* 03:17 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:17 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: dde1dc09-44b5-42af-ac49-{{Gerrit|c58dbbae7cb4}}, use this to unset). - cookbook ran by andrew@buster
* 03:14 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: b59fa99b-8713-4e63-8c56-{{Gerrit|08bd71fdfbe3}}, use this to unset). - cookbook ran by andrew@buster
* 03:13 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:07 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 3547c843-6b15-4151-b452-{{Gerrit|6f618a886b7b}}, use this to unset). - cookbook ran by andrew@buster
* 03:07 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 6f5808d0-afb2-4222-ae8e-{{Gerrit|4d953d49cd63}}, use this to unset). - cookbook ran by andrew@buster
* 03:06 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: da738ba3-883f-4d76-bb33-{{Gerrit|7eb551de5fc2}}, use this to unset). - cookbook ran by andrew@buster
* 03:05 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:05 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:04 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:04 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:03 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f90ca4c1-78af-46bd-9262-{{Gerrit|0fa4edc2898e}}, use this to unset). - cookbook ran by andrew@buster
* 03:02 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:00 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:00 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 62383663-9ae6-4a15-a2a6-{{Gerrit|62be06adbc96}}, use this to unset). - cookbook ran by andrew@buster
* 02:59 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:59 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:59 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 6e0da344-49ad-451f-8b8d-{{Gerrit|ea8f853a6f89}}, use this to unset). - cookbook ran by andrew@buster
* 02:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:52 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: c9ec5b7c-aa9f-4afd-a417-{{Gerrit|54c52c07ca47}}, use this to unset). - cookbook ran by andrew@buster
* 02:51 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:51 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:48 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 5509c648-58f6-49fd-90e8-{{Gerrit|ac9aa66e8b04}}, use this to unset). - cookbook ran by andrew@buster
* 02:48 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:48 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: af4ba9ad-e765-4087-86aa-{{Gerrit|9111c0821175}}, use this to unset). - cookbook ran by andrew@buster
* 02:46 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:45 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:45 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: 6da98123-8838-4d2e-8659-{{Gerrit|d769bfa292fe}}, use this to unset). - cookbook ran by andrew@buster
* 02:44 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:44 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:43 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 02:43 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:42 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:42 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 91860079-8fff-40db-9de7-{{Gerrit|17741d528ad6}}, use this to unset). - cookbook ran by andrew@buster
* 02:41 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:41 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:39 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: a6a0f414-d305-4a09-87d2-{{Gerrit|066eec83642b}}, use this to unset). - cookbook ran by andrew@buster
* 02:39 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 6000898c-4e5e-49c8-8841-{{Gerrit|3262d6da0366}}, use this to unset). - cookbook ran by andrew@buster
* 02:37 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:37 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 24a8b50e-5c04-40e6-a732-{{Gerrit|f0232b83e240}}, use this to unset). - cookbook ran by andrew@buster
* 02:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8d259aca-1803-4346-b5a4-{{Gerrit|a447673b3537}}, use this to unset). - cookbook ran by andrew@buster
* 02:34 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:33 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 704aca7f-0069-4a4a-b8de-{{Gerrit|1aba7b9ca7ef}}, use this to unset). - cookbook ran by andrew@buster
* 02:32 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04c0fba3-e2cb-40b4-abfd-{{Gerrit|d24165c9bd55}}, use this to unset). - cookbook ran by andrew@buster
* 02:32 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 25e5e0c8-5c27-460e-9b44-{{Gerrit|1cbd1c2fb551}}, use this to unset). - cookbook ran by andrew@buster
* 02:28 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:25 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5e62968a-0bb6-43b1-ae5e-{{Gerrit|e352442ef976}}, use this to unset). - cookbook ran by andrew@buster
* 02:24 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:24 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:21 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5b1a8e8f-8699-433d-b764-{{Gerrit|197862dc5cf1}}, use this to unset). - cookbook ran by andrew@buster
* 02:20 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:20 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:15 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:15 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:13 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 299c466e-602b-4a8e-bb37-{{Gerrit|156553dbe426}}, use this to unset). - cookbook ran by andrew@buster
* 02:13 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:13 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:12 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:12 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:03 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6a64d619-c484-411f-ad9d-{{Gerrit|89d43b063737}}, use this to unset). - cookbook ran by andrew@buster
* 02:02 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:02 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 301309a2-a8b5-4698-98d6-{{Gerrit|bb4aa0a75e45}}, use this to unset). - cookbook ran by andrew@buster
* 00:46 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: b2f8c6a2-7224-4fb1-8e12-{{Gerrit|dff6dda02380}}, use this to unset). - cookbook ran by andrew@buster
* 00:46 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:40 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:40 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:38 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: c0f62e0e-7865-4e77-a238-{{Gerrit|d707ceed53a8}}, use this to unset). - cookbook ran by andrew@buster
* 00:38 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:38 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:36 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 96a368d5-232f-4187-b0bb-{{Gerrit|5923a50fff71}}, use this to unset). - cookbook ran by andrew@buster
* 00:36 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:36 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:35 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 31b6bfcb-81cb-4d89-b977-{{Gerrit|190899ea2e64}}, use this to unset). - cookbook ran by andrew@buster
* 00:35 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:35 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:28 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 50f763d4-36ea-4241-8e71-{{Gerrit|cc6aa20ccc9e}}, use this to unset). - cookbook ran by andrew@buster
* 00:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: cd3d17af-ee15-4a9f-8750-{{Gerrit|66b97a46ea12}}, use this to unset). - cookbook ran by andrew@buster
* 00:27 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 9b075166-d424-42c0-8e06-{{Gerrit|f74a47abb798}}, use this to unset). - cookbook ran by andrew@buster
* 00:21 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-07-18 ===
* 23:11 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: e0aab64f-d911-4d7e-9a97-{{Gerrit|c59d9868ceea}}, use this to unset). - cookbook ran by andrew@buster
* 23:11 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 447d9c7a-a00e-41db-93ed-{{Gerrit|6d2bfc66e5df}}, use this to unset). - cookbook ran by andrew@buster
* 23:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:01 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 46c2ddf2-50c3-4824-ba72-{{Gerrit|d7fb3fa4c5e0}}, use this to unset). - cookbook ran by andrew@buster
* 23:01 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6facdf4f-7bb8-42cd-8ac8-{{Gerrit|b4412fc4cf70}}, use this to unset). - cookbook ran by andrew@buster
* 23:00 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:52 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 23190573-fc0d-4872-8343-{{Gerrit|e35b84132028}}, use this to unset). - cookbook ran by andrew@buster
* 22:51 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:51 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:51 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 990313c4-5fa6-4de7-a8bf-{{Gerrit|b361f0c79e90}}, use this to unset). - cookbook ran by andrew@buster
* 22:50 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:50 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:48 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:48 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:42 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 59367d39-bd2f-47fd-becd-{{Gerrit|255bf823ff12}}, use this to unset). - cookbook ran by andrew@buster
* 22:42 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:42 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:40 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 38602815-d187-4455-9676-{{Gerrit|59607505bba7}}, use this to unset). - cookbook ran by andrew@buster
* 22:39 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 00f21e1b-d014-4bc5-995f-{{Gerrit|e25201fc77c4}}, use this to unset). - cookbook ran by andrew@buster
* 22:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:31 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: d05d07b2-8c60-46de-a89e-{{Gerrit|76987901901b}}, use this to unset). - cookbook ran by andrew@buster
* 22:31 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 47f6b9e6-ceab-4fb6-8f76-{{Gerrit|0de4341d0841}}, use this to unset). - cookbook ran by andrew@buster
* 22:31 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:31 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 599042a5-da25-4985-b502-{{Gerrit|328fe40a72d5}}, use this to unset). - cookbook ran by andrew@buster
* 22:29 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:29 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:29 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: a554d107-c14d-41f8-b180-{{Gerrit|093e2cf73e72}}, use this to unset). - cookbook ran by andrew@buster
* 22:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9fef70c7-6f7d-4b33-a9bc-{{Gerrit|8c39630dc182}}, use this to unset). - cookbook ran by andrew@buster
* 22:26 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:02 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 3744c0fa-084d-4198-98ca-{{Gerrit|66c7a5210f41}}, use this to unset). - cookbook ran by andrew@buster
* 22:02 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 01caf1e1-df64-474a-94d5-{{Gerrit|f37751e23d62}}, use this to unset). - cookbook ran by andrew@buster
* 22:01 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:59 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 3018c2fc-9cd5-45c8-a160-{{Gerrit|d5ad5728c8d5}}, use this to unset). - cookbook ran by andrew@buster
* 21:59 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:59 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: d760f916-32ea-4194-8974-{{Gerrit|f36064a9866c}}, use this to unset). - cookbook ran by andrew@buster
* 21:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:00 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: cb2a38f3-e9ca-4bf6-989c-{{Gerrit|a27be2b2d88c}}, use this to unset). - cookbook ran by andrew@buster
* 20:59 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:58 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 93fa9477-54b8-46e7-8508-{{Gerrit|a2d492528d1f}}, use this to unset). - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f6cf5b29-ce8b-426e-af1c-{{Gerrit|cab74e0c4a3a}}, use this to unset). - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: fe26e612-d76e-4120-b60a-{{Gerrit|4f300add224f}}, use this to unset). - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 2abd94be-b6d8-4042-a9ce-{{Gerrit|998e41d932f4}}, use this to unset). - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 10973c71-30d8-44bf-a310-{{Gerrit|d0c1af76d399}}, use this to unset). - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f8f1b214-ec76-4157-a8e6-{{Gerrit|6374e5d14608}}, use this to unset). - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:11 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 95dae613-2b04-472e-a8f9-{{Gerrit|e71c3fbb8d0e}}, use this to unset). - cookbook ran by andrew@buster
* 20:10 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: c804f4db-ea63-44e9-aff8-{{Gerrit|fea54c238345}}, use this to unset). - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 597a901d-4624-43de-b986-{{Gerrit|c4c6c2fcf80c}}, use this to unset). - cookbook ran by andrew@buster
* 20:03 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:03 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: fdceed0d-6fa0-4806-8fb6-{{Gerrit|3b321a5a1623}}, use this to unset). - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:37 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: b0904345-9aa3-4202-b992-{{Gerrit|78644141517c}}, use this to unset). - cookbook ran by andrew@buster
* 19:36 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:31 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:31 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-07-13 ===
* 14:48 bd808: Added Tavvi as member of #acl*wmcs-team

=== 2022-07-12 ===
* 12:04 wm-bot2: Ceph cluster at &lt;nowiki&gt;{&lt;/nowiki&gt;self.deployment&lt;nowiki&gt;}&lt;/nowiki&gt; set out of maintenance. - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Set the ceph cluster for codfw1dev in maintenance, alert silence ids: db32805f-a033-4e0e-8fc7-{{Gerrit|ed0c2e9f8be1}},f4e698f0-4b51-4b07-9ba8-{{Gerrit|0b296ca1c4fb}},39ad5325-44ed-44d1-bba5-{{Gerrit|91a85c7a8401}},a584a3c6-9dae-41e9-ac82-{{Gerrit|a91cd35af8fb}} - cookbook ran by dcaro@vulcanus
* 08:37 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2005-dev', 'cloudnet2006-dev'] - cookbook ran by dcaro@vulcanus
* 08:37 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:31 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2005-dev,cloudnet2006-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-08 ===
* 15:57 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-07-07 ===
* 07:17 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T312509|T312509]]) - cookbook ran by dcaro@vulcanus
* 07:12 wm-bot2: Rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T312509|T312509]]) - cookbook ran by dcaro@vulcanus

=== 2022-07-06 ===
* 17:50 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: ['8a5b9eee-48c0-474d-8277-faeb05a2ea61', '65aad0fc-d887-47a3-b20c-d1ed461a2411', '86b078ae-3a27-4063-8c7c-198a2fe0c172'] - cookbook ran by dcaro@vulcanus

=== 2022-07-04 ===
* 13:27 wm-bot2: Rebooting cloudgw host cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot2: Rebooted cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Rebooting cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Rebooting all the cloudgw nodes from the eqiad1 deployment: cloudgw1001.eqiad.wmnet,cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:13 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 13:12 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:09 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:56 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:52 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:56 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 10:56 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:51 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:51 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:47 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:47 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:41 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 09:07 wm-bot2: Rebooting cloudnet host cloudnet1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:06 wm-bot2: Rebooted cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot2: Rebooting cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot2: Rebooting all the cloudnet nodes cloudnet1003,cloudnet1004 - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:53 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:53 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:47 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:47 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:55 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:44 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 07:44 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:36 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:36 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:09 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:05 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:04 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-03 ===
* 21:27 andrewbogott: rebuilding rabbit cluster in codfw1dev to get rid of some queues so unresponsive that they can't otherwise be deleted

=== 2022-07-02 ===
* 11:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-01 ===
* 15:35 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:30 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:29 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:25 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:42 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 14:42 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:32 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:02 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:58 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:58 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:54 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:52 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:37 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:34 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:30 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-06-30 ===
* 18:17 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 18:13 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:45 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 06:52 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-06-29 ===
* 08:45 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 08:41 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-06-28 ===
* 13:03 taavi: grant the tools project access to the g3.cores16.ram64.disk20.10xiops flavor [[phab:T301949|T301949]]

=== 2022-06-22 ===
* 16:48 taavi: restart designate-*.service on both cloudservices nodes
* 16:44 taavi: restart nova-conductor on all the cloudcontrol nodes
* 12:50 andrewbogott: rebooting each eqiad1 cloudcontrol node in hopes of getting a baseline re: openstack instability

=== 2022-06-21 ===
* 04:48 andrewbogott: stopping nova-fullstack agent on cloudcontrol1003; it's going to page us otherwise and we're all AFK tomorrow
* 04:02 andrewbogott: restarting rabbitmq on cloudcontrol100x (one at a time)

=== 2022-06-17 ===
* 17:53 andrewbogott: switching to a new python-based health check for galera and haproxy. This may make things more stable, or it may not. [[phab:T310664|T310664]]
* 06:15 taavi: restart neutron-linuxbridge-agent on cloudvirt1046

=== 2022-06-15 ===
* 11:34 taavi: restart neutron-linuxbridge-agent on cloudvirt1022

=== 2022-06-14 ===
* 16:26 wm-bot2: OSDs (['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloudcephosd1021', 'cl
* 14:38 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudceph
* 12:55 wm-bot2: OSDs (['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 12:44 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus

=== 2022-06-13 ===
* 11:14 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:07 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:05 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:04 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:03 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 09:15 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus

=== 2022-06-06 ===
* 13:21 andrewbogott: restarting mysql/galera on cloudcontrol100x in an attempt to stabilize some flapping there

=== 2022-06-02 ===
* 07:51 taavi: restart neutron-linuxbridge-agent.service on cloudvirt1034 [[phab:T309732|T309732]]
* 00:37 andrewbogott: updated nameservers for codfw1dev instances via 'openstack subnet set --dns-nameserver etc.'

=== 2022-06-01 ===
* 17:11 andrewbogott: restarting designate services in cloudservices1xxx hosts
* 16:37 taavi: root@cloudcontrol1005:~# cinder reset-state --state available 491066a4-16a9-4ce4-a9f6-{{Gerrit|182660616d77}} for [[phab:T309659|T309659]]

=== 2022-05-30 ===
* 17:43 andrewbogott: restarting neutron-rpc and neutron-api services on cloudcontrol1xxx

=== 2022-05-29 ===
* 14:55 andrewbogott: restarting nova services on all eqiad1 cloudcontrol nodes to recover from rabbit breakage
* 14:15 andrewbogott: restarting rabbitmq on all eqiad1 cloudcontrol nodes (one at a time)

=== 2022-05-25 ===
* 20:01 balloons: clean up cinder backup a bit, restart service due to network outage
* 20:01 balloons: cloudvirt1029 restarted nova due to network outage

=== 2022-05-19 ===
* 15:21 andrewbogott: resetting password for the 'troveguest' rabbitmq user. I think I may have broken this during a recent rebuild of the rabbitmq cluster

=== 2022-05-18 ===
* 15:42 andrewbogott: updated the 'debian-11.0-bullseye' glance image with a fresh build

=== 2022-05-14 ===
* 11:33 taavi: deleted projects 'ores' and 'ores-staging' [[phab:T308102|T308102]]

=== 2022-05-13 ===
* 06:20 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 06:20 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:16 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:16 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:15 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:15 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:11 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 06:11 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:10 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:10 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:09 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:07 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:06 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:06 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:05 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:51 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:50 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:50 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:47 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:47 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:44 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:44 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:41 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:40 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:40 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:38 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:37 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:37 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:30 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:29 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:29 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:19 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:12 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:12 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:08 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:01 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:52 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:48 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:48 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:44 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:31 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:24 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:21 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:21 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:18 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:37 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:36 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:36 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:34 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:34 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:27 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:27 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:25 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:25 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:55 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:55 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:04 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:23 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 01:23 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:19 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:01 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:01 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:25 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 00:25 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:20 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:19 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:19 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:11 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 00:11 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:07 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-12 ===
* 23:55 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:55 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:23 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:23 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:20 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:17 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:17 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:13 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:57 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:56 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:56 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:55 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:55 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:54 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:54 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:53 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:52 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:52 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:51 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:44 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:44 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:42 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:36 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:36 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:33 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:33 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:30 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:22 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:22 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:21 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:21 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:11 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:11 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:08 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:08 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:07 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:07 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:05 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:04 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:04 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:00 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:55 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:55 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:53 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:50 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:48 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:48 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:44 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:44 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:43 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:43 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:31 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:23 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:22 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:22 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:11 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:07 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:00 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:58 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:35 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:35 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:06 andrewbogott: stopping nfs-server on labstore1004 in preparation for reboot
* 04:12 andrewbogott: rebooting primary bastion (bastion-eqiad1-03.bastion.eqiad1.wikimedia.cloud) in hopes of resolving a problem with ssh proxying

=== 2022-05-11 ===
* 18:48 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:39 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:38 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:38 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:04 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:03 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:03 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 08:56 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 08:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 07:53 dcaro: test
* 04:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:43 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:43 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:39 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:23 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:09 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:08 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:08 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:04 andrewbogott: reset and recreated the rabbitmq cluster in eqiad1 to get around some broken queues.
* 03:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:25 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:25 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-10 ===
* 21:43 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:40 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:40 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:35 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:32 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:05 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:02 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:55 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:55 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:47 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:46 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:45 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:45 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:40 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:39 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:39 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:37 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:36 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:33 wm-bot: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 19:33 wm-bot: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:29 wm-bot: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:06 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:05 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:05 wm-bot: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:41 andrewbogott: rebooting cloud*-dev for [[phab:T307668|T307668]]
* 13:59 taavi: manually attached [[User:Dreamy Jazz]] to wikitech for a password reset (https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Manually_associate_an_LDAP_account_with_wikitech)

=== 2022-05-07 ===
* 01:33 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:32 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:30 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:21 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:18 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-03 ===
* 20:38 andrewbogott: upgrading clouddb2001-dev in place
* 18:18 taavi: updated 'puppet-enc' endpoints on the keystone catalog to use https and port 443

=== 2022-05-02 ===
* 16:56 dcaro: rebooting cloudmetrics1001

=== 2022-04-29 ===
* 14:22 andrewbogott: changing login.toolforge.org, bastion.toolforge.org, and dev.toolforge.org dns entries to refer to the new Buster bastions [[phab:T277653|T277653]] https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Timeline

=== 2022-04-27 ===
* 14:51 wm-bot: Finished rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloud
* 14:50 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:41 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:41 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot: Finished rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot: Rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot: Finished rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:01 wm-bot: Rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:01 wm-bot: Finished rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Finished rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:44 wm-bot: Rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:43 wm-bot: Finished rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:39 wm-bot: Rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:39 wm-bot: Finished rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:35 wm-bot: Rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:35 wm-bot: Finished rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Finished rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:26 wm-bot: Rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:26 wm-bot: Finished rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot: Rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot: Finished rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot: Rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot: Finished rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot: Rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot: Finished rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Finished rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:57 wm-bot: Rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:57 wm-bot: Finished rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Finished rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Finished rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:46 wm-bot: Rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:46 wm-bot: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024 - cookbo
* 12:15 wm-bot: Finished rebooting the nodes ['cloudcephmon1001', 'cloudcephmon1002', 'cloudcephmon1003'] - cookbook ran by dcaro@vulcanus
* 12:15 wm-bot: Finished rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:12 wm-bot: Rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:12 wm-bot: Finished rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:09 wm-bot: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:09 wm-bot: Finished rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:07 wm-bot: Rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:07 wm-bot: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:02 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:02 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:59 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:59 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 11:55 wm-bot: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] - cookbook ran by dcaro@vulcanus
* 11:55 wm-bot: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot: Rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:47 wm-bot: Rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:47 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:43 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:43 wm-bot: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev - cookbook ran by dcaro@vulcanus

=== 2022-04-26 ===
* 10:36 taavi: [codfw1dev] updated designate pool to 2004/2005-dev according to the instructions on https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS/Designate#Initial_designate/pdns_node_setup

=== 2022-04-22 ===
* 10:33 taavi: [codfw1dev] restart designate-sink on both new cloudservices host to fix rabbitmq connectivity

=== 2022-04-21 ===
* 05:38 andrewbogott: replaced cloudservices200[2,3] with cloudservices200[4,5]

=== 2022-04-19 ===
* 15:29 andrewbogott: stopping all VMs on cloudvirt1019, reimaging host

=== 2022-04-18 ===
* 15:23 andrewbogott: reimaging cloudvirt1020, leaving VMs in place
* 13:40 andrewbogott: shutting down many codfdfw1dev servers (including network infra!) for [[phab:T305469|T305469]]

=== 2022-04-14 ===
* 20:14 andrewbogott: restarting nova-api and nova-conductor services in a superstitious attempt to reduce open DB connections

=== 2022-04-13 ===
* 22:01 andrewbogott: restarting galera on cloudcontrols (one by one) to clear open connections

=== 2022-04-11 ===
* 15:59 taavi: created cloudinfra.wmcloud.org zone

=== 2022-04-09 ===
* 19:55 andrewbogott: reimaging cloudbackup1001-dev to bullseye
* 19:37 taavi: add 'puppet-enc' service &amp; endpoint to keystone [[phab:T274666|T274666]]
* 19:25 andrewbogott: reimaging cloudbackup1002-dev to bullseye

=== 2022-04-07 ===
* 12:51 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T305631|T305631]]) - cookbook ran by arturo@nostromo

=== 2022-04-06 ===
* 09:12 arturo: [codf1dev] installing python3-eventlet 0.30.2-5~bpo11+1 on all required servers (cloudvirt, cloudnet, cloudcontrol) ([[phab:T305157|T305157]])
* 08:45 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudvirt2003-dev ([[phab:T305157|T305157]])
* 08:42 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudcontrol servers ([[phab:T305157|T305157]])
* 08:24 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudvirt2003-dev ([[phab:T305157|T305157]])
* 08:20 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudcontrol servers ([[phab:T305157|T305157]])

=== 2022-03-30 ===
* 11:20 arturo: apply urpf strict filter to eqiad cloud-hosts vlan - [[phab:T285461|T285461]]

=== 2022-03-29 ===
* 10:02 dcaro: restarting keystone ([[phab:T304918|T304918]])

=== 2022-03-23 ===
* 22:53 wm-bot: Drained 'cloudvirt1045.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:38 wm-bot: Drained 'cloudvirt1044.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:12 wm-bot: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:12 wm-bot: Draining 'cloudvirt1045.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:08 wm-bot: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:07 wm-bot: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:06 wm-bot: Draining 'cloudvirt1044.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:06 wm-bot: Draining 'cloudvirt1043.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:54 wm-bot: Drained 'cloudvirt1042.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:19 wm-bot: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:19 wm-bot: Draining 'cloudvirt1042.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:12 wm-bot: Drained 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:12 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:09 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:07 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:04 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:55 wm-bot: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:54 wm-bot: Draining 'cloudvirt1041.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:30 wm-bot: Drained 'cloudvirt1039.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:14 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:14 wm-bot: Draining 'cloudvirt1039.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:44 wm-bot: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:43 wm-bot: Draining 'cloudvirt1038.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:19 wm-bot: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:18 wm-bot: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:13 wm-bot: Drained 'cloudvirt1036.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:02 wm-bot2: Testing wm-bot relay to #wikimedia-cloud-feed
* 17:55 wm-bot: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:54 wm-bot: Draining 'cloudvirt1036.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:04 wm-bot: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:03 wm-bot: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:03 wm-bot: Drained 'cloudvirt1034.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:51 wm-bot: Drained 'cloudvirt1033.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:37 wm-bot: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:37 wm-bot: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Draining 'cloudvirt1034.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 15:01 wm-bot: Drained 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 15:00 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:57 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:44 wm-bot: Drained 'cloudvirt1031.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:35 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:34 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:32 wm-bot: Drained 'cloudvirt1030.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:20 wm-bot: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:19 wm-bot: Draining 'cloudvirt1031.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:18 wm-bot: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:17 wm-bot: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:54 taavi: restart nova-fullstack on cloudcontrol1003 to pick up bastion ip change
* 13:43 wm-bot: Drained 'cloudvirt1029.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:23 wm-bot: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:22 wm-bot: Draining 'cloudvirt1029.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-22 ===
* 22:59 wm-bot: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:58 wm-bot: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-17 ===
* 01:09 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:53 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:52 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:52 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-15 ===
* 20:58 wm-bot: Drained 'cloudvirt1026.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:14 wm-bot: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 13:14 wm-bot: Unsetting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 10:32 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 10:30 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo

=== 2022-03-14 ===
* 21:24 wm-bot: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:59 wm-bot: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:58 wm-bot: Setting cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:58 wm-bot: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:02 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:59 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:59 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:16 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:13 wm-bot: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:56 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:55 wm-bot: Setting cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:55 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:53 wm-bot: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:52 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:51 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:51 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:50 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 11:48 dcaro: rebased cookbooks on latest master, make sure you pull before sending new patches

=== 2022-03-08 ===
* 18:29 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:29 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:29 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:23 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:21 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:21 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:18 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:17 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:17 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:27 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:27 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:18 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:15 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:15 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:47 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:47 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:08 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:07 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:07 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:11 arturo: [codfw1dev] rebooting cloudservices servers for [[phab:T303179|T303179]]
* 13:07 arturo: [codfw1dev] rebooting cloudvirt servers for [[phab:T303179|T303179]]
* 13:06 arturo: [codfw1dev] rebooting cloudnet servers for [[phab:T303179|T303179]]
* 12:55 arturo: [codfw1dev] rebooting cloudcontrol servers for [[phab:T303179|T303179]]

=== 2022-03-03 ===
* 08:49 taavi: deploying cloudmetrics grafana to grafana 8, [[phab:T282863|T282863]]

=== 2022-03-02 ===
* 09:06 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/701347

=== 2022-02-28 ===
* 15:30 dcaro: cleaning up leftover snapshots from failed backups of the maps volume ([[phab:T302720|T302720]])

=== 2022-02-24 ===
* 17:04 andrewbogott: upgrading eqiad1 and codfw1dev to mariadb 10.5.15+maria~bullseye via 'apt-get install libmariadb3:amd64 galera-4 mariadb-server'
* 15:42 dcaro: stopping and starting mariadb on cloudcontrol1003 ([[phab:T302146|T302146]])
* 10:37 arturo: [codfw1dev] briefly installed galera-4 (26.4.11+1bullseye) over (26.4.9-0+deb11u1) on cloudcontrol2001-dev and then downgrade again to verify package install ([[phab:T302482|T302482]])

=== 2022-02-23 ===
* 20:39 taavi: added domain-wide 'designateadmin' and 'observer' roles to project-proxy-dns-manager service account [[phab:T295246|T295246]]
* 17:40 andrewbogott: restarting lots of openstack services to try to clear up the mess that is [[phab:T236101|T236101]]
* 12:13 arturo: cleaning up cinder volume snapshots, aborrero@cloudcontrol1005:~$ for i in $(sudo wmcs-openstack volume snapshot list -f value -c ID) ; do sudo wmcs-openstack volume snapshot delete $i ; done ([[phab:T302382|T302382]])
* 10:14 arturo: cleaning up neutron agents for non-existent servers cloudvirt100[1-9].eqiad.wmnet,cloudvirt10[12-15].eqiad.wmnet
* 10:05 dcaro: Deleting stuck novafullstack servers, to let the service create new ones ([[phab:T302369|T302369]])
* 09:56 arturo: neutron agent-delete bad663b3-fd25-4393-a546-{{Gerrit|4b1b4bdec4db}} (Linux bridge agent {{!}} cloudvirtan1001)
* 09:56 arturo: neutron agent-delete 1071c198-ed57-4b5a-9439-{{Gerrit|30e66a31aa69}} (Linux bridge agent {{!}} cloudvirtan1005)
* 09:55 arturo: neutron agent-delete 2eeef198-8af7-4e5d-bd73-{{Gerrit|e14a2a8d2404}} (Linux bridge agent {{!}} cloudvirtan1004)
* 09:55 arturo: neutron agent-delete afe173eb-35ba-444a-9960-{{Gerrit|899629786d2f}} (Linux bridge agent {{!}} cloudvirtan1003)
* 09:54 arturo: neutron agent-delete afcb9b7f-c1a6-4ff4-9b10-{{Gerrit|92bfbe8d1a56}} (Linux bridge agent {{!}} cloudvirtan1002)
* 09:39 dcaro: restarting neutron-api cloudcontrol1003 to see if the agent status update starts working ([[phab:T302369|T302369]])
* 09:38 dcaro: restarting neutron-dhcp-agent on cloudnet1003 ([[phab:T302369|T302369]])

=== 2022-02-22 ===
* 22:10 andrewbogott: raising project 'maps' quota by two tb -- [[phab:T300160|T300160]]
* 09:24 arturo: restarting mariadb @ cloudcontrol1003 ([[phab:T302146|T302146]])
* 09:13 arturo: restarting mariadb @ cloudcontrol1004 ([[phab:T302146|T302146]])

=== 2022-02-18 ===
* 21:57 andrewbogott: leaving cloudcontrol1003 downtimed with disabled puppet for the weekend.  Everything there should be stable and fine save rabbit which needs an upgrade.
* 21:30 andrewbogott: rebooting cloudcontrol1003 because rabbit is freaking out
* 17:25 andrewbogott: in-place upgrade of cloudcontrol1004 to bullseye -- [[phab:T281276|T281276]]
* 12:34 arturo: manually install prometheus-openstack-exporter on cloudcontrol1005 ([[phab:T302050|T302050]])

=== 2022-02-17 ===
* 23:02 andrewbogott: in-place upgrade to Bullseye on cloudcontrol1005 [[phab:T281276|T281276]]

=== 2022-02-15 ===
* 14:15 taavi: [codfw1dev] added domain-wide 'designateadmin' and 'observer' roles to codfw1dev-proxy-dns-manager service account [[phab:T295246|T295246]]

=== 2022-02-04 ===
* 10:12 arturo: restart backup_vms service in cloudvirt1024 ([[phab:T300956|T300956]])

=== 2022-02-03 ===
* 08:21 taavi: cloudmetrics1004: manually added an empty line to /etc/prometheus/blackbox.yml to make /usr/local/bin/blackbox-exporter-assemble happy (clearing &quot;performing a change every puppet run&quot; alert)

=== 2022-02-02 ===
* 02:36 andrewbogott: restarting mariadb on cloudcontrol1004

=== 2022-01-31 ===
* 10:15 arturo: cloudcontrol1005:~$ sudo systemctl restart backup_glance_images.service (failed state, no logs, icinga alert)

=== 2022-01-29 ===
* 18:24 taavi: delete 2 puppet prefixes in a weird state [[phab:T299750|T299750]]

=== 2022-01-27 ===
* 13:24 arturo: cloudmetrics1004:~ $ sudo systemctl restart wmcs_monitoring_graphite_rsync.service ([[phab:T300138|T300138]])

=== 2022-01-26 ===
* 19:09 andrewbogott: bootstrapping a fresh galera node on cloudcontrol1004
* 18:57 andrewbogott: restarting mariadb on cloudcontrol1004

=== 2022-01-25 ===
* 10:49 arturo: made cloudmetrics1001/1002 primary/backup respectively ([[phab:T299744|T299744]], [[phab:T297814|T297814]], [[phab:T300011|T300011]])

=== 2022-01-19 ===
* 16:38 andrewbogott: moving all scratch mounts to scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud

=== 2022-01-05 ===
* 03:11 andrewbogott: 'cp /etc/apt/sources.list /etc/apt/sources.list.prepuppet' on all VMs. Backing up state before puppetizing sources.list with https://gerrit.wikimedia.org/r/c/operations/puppet/+/751498

=== 2022-01-04 ===
* 12:44 dcaro: increasing the size_limit for labs ldap servers

=== 2021-12-26 ===
* 16:55 majavah: run attachLdapUser.php on wikitech for developer account &quot;Karthiksripal&quot;

=== 2021-12-24 ===
* 22:51 majavah: ran the wikireplica dns script on s5 [[phab:T298303|T298303]]

=== 2021-12-23 ===
* 21:42 majavah: deployed horizon wmf-proxy-dashboard update to fix editing of existing proxies

=== 2021-12-21 ===
* 10:39 arturo: dropped egress NAT exceptions for WMF apt repos, [[phab:T298042|T298042]]

=== 2021-12-15 ===
* 12:44 dcaro: Downtiming cloudvirt-wdqs1001 as it has no VMs running until disk space is fixed ([[phab:T297454|T297454]])

=== 2021-12-14 ===
* 10:26 dcaro: Moved the nova cache (/var/lib/nova/instances/_base) and the canary image local data (/var/lib/nova/instance/&lt;canary_image_id&gt;) to the root disk on cloudvirt-wdqs1001 to temporary free some space ([[phab:T297454|T297454]])

=== 2021-12-13 ===
* 18:08 wm-bot: Drained 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:50 wm-bot: Set cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:49 wm-bot: Setting cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:49 wm-bot: Draining 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:44 wm-bot: Drained 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:30 wm-bot: Set cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:30 wm-bot: Setting cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:30 wm-bot: Draining 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:13 wm-bot: Drained 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
* 16:50 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:47 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:47 wm-bot: Draining 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
* 16:44 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:43 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse

=== 2021-12-03 ===
* 18:56 andrewbogott: maintain-views and maintain-meta-p on clouddb1013-1020
* 10:49 majavah: deleting dbbackups-dashboard project [[phab:T296992|T296992]]

=== 2021-12-02 ===
* 01:17 wm-bot: Drained 'cloudvirt1028.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:56 wm-bot: Set cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:56 wm-bot: Setting cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:56 wm-bot: Draining 'cloudvirt1028.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:50 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:50 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:50 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:28 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:03 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:02 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:02 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster

=== 2021-12-01 ===
* 23:59 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:59 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 23:54 andrewbogott: *correction* adding spare cloudvirts 1044 and 1045 to the 'ceph' pool in order to make space for future juggling around [[phab:T296790|T296790]] and [[phab:T296792|T296792]]
* 23:53 andrewbogott: adding spare cloudvirts 1044 and 1055 to the 'ceph' pool in order to make space for future juggling around [[phab:T296790|T296790]] and [[phab:T296792|T296792]]

=== 2021-11-28 ===
* 17:48 andrewbogott: moved cloudvirt1018 out of the 'localstorage' aggregate and into 'maintenance' for [[phab:T296592|T296592]].  It will need to be moved back after the raid is rebuilt.

=== 2021-11-21 ===
* 07:19 dcaro_away: restarting designate-sink with some extra logs in it ([[phab:T296144|T296144]])

=== 2021-11-17 ===
* 15:48 andrewbogott: upgrading mariadb packages on eqiad1 cloudcontrols
* 15:39 andrewbogott: sudo cumin  &quot;cloud*&quot; 'apt-get update -y --allow-releaseinfo-change'
* 15:26 andrewbogott: updated mariadb packages on codfw1dev cloudcontrols to 1:10.3.31-0+deb10u1

=== 2021-11-12 ===
* 13:31 arturo: restarting glance-api services to make sure they work with new ceph auth creds ([[phab:T293752|T293752]])

=== 2021-11-08 ===
* 21:50 andrewbogott: returned clouddb pools back to normal after maintain_views run: https://gerrit.wikimedia.org/r/c/operations/puppet/+/737505 [[phab:T216481|T216481]]
* 20:07 andrewbogott: depooling clouddb1013 for maintain_views attempt
* 10:54 arturo: [codfw1dev] create service account `srv-networktests` following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for [[phab:T294955|T294955]]
* 10:34 arturo: create service account `srv-networktests` following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for [[phab:T294955|T294955]]

=== 2021-11-05 ===
* 11:18 wm-bot: Added 1 new OSDs ['cloudcephosd1024.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:17 wm-bot: Added OSD cloudcephosd1024.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:15 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:12 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:12 wm-bot: Adding OSD cloudcephosd1024.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:12 wm-bot: Adding new OSDs ['cloudcephosd1024.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance

=== 2021-11-04 ===
* 16:39 wm-bot: Added 1 new OSDs ['cloudcephosd1023.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:39 wm-bot: Added OSD cloudcephosd1023.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:37 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:34 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:33 wm-bot: Adding OSD cloudcephosd1023.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:33 wm-bot: Adding new OSDs ['cloudcephosd1023.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:17 wm-bot: Added 1 new OSDs ['cloudcephosd1022.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:17 wm-bot: Added OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:16 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:13 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:12 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:12 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:00 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:00 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:26 wm-bot: Added 1 new OSDs ['cloudcephosd1021.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:26 wm-bot: Added OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:23 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:20 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:19 wm-bot: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:19 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:16 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance

=== 2021-11-03 ===
* 17:22 arturo: [codfw1dev] installing keepalived 2.1.5 from buster-backports on cloudgw2001-dev/2002-dev ([[phab:T294956|T294956]])
* 11:45 arturo: [codfw1dev] downgrade kernel on cloudgw2001-dev/2002-dev ([[phab:T294853|T294853]], [[phab:T291813|T291813]])

=== 2021-11-02 ===
* 10:54 arturo: rebooting cloudnet1004/1003 for [[phab:T291813|T291813]]
* 10:43 arturo: [codfw1dev] rebooting cloudgw200[12]-dev for [[phab:T291813|T291813]]

=== 2021-10-24 ===
* 00:47 andrewbogott: deploying a change so that openstack clients use tls endpoints: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732738

=== 2021-10-21 ===
* 10:19 arturo: drop firewall exception on core routers for wiki replicas legacy setup ([[phab:T293897|T293897]])
* 10:12 arturo: drop NAT exception for wiki replicas legacy setup ([[phab:T293897|T293897]])

=== 2021-10-20 ===
* 21:06 andrewbogott: creating cloudinfra-nfs project [[phab:T293936|T293936]]

=== 2021-10-18 ===
* 19:21 andrewbogott: also ticked the 'admin' box on wikitech for majavah [[phab:T292827|T292827]]
* 18:58 andrewbogott: granting majavah 'admin' role in the 'admin' project and also in the default domain.  [[phab:T292827|T292827]]

=== 2021-10-14 ===
* 12:28 arturo: [codfw1dev] add DB grants for cloudbackup2002.codfw.wmnet IP address to the cinder DB ([[phab:T292546|T292546]])

=== 2021-10-13 ===
* 10:46 arturo: updating python3-neutron across the fleet ([[phab:T292936|T292936]])

=== 2021-10-12 ===
* 09:06 dcaro: upgrading eqiad cloudnet hosts neutron packages ([[phab:T292936|T292936]])
* 08:57 dcaro: upgrading codfw cloudnet hosts neutron packages ([[phab:T292936|T292936]])

=== 2021-10-05 ===
* 09:39 arturo: [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) [[phab:T291257|T291257]]

=== 2021-09-30 ===
* 14:50 andrewbogott: sudo cumin  &quot;cloud*&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot; and sudo cumin  &quot;lab*&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot;  [[phab:T292202|T292202]]
* 14:43 andrewbogott: ran sudo cumin --force --timeout 500 -o json  &quot;A:all&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot; to get nslcd happy again [[phab:T292202|T292202]]

=== 2021-09-29 ===
* 09:41 arturo: [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured ([[phab:T291257|T291257]])

=== 2021-09-28 ===
* 16:23 bstorm: downtime for clouddb1020 to reduce re-pages in case this goes badly [[phab:T291963|T291963]]
* 16:21 bstorm: powering on clouddb1020 via remote console [[phab:T291963|T291963]]
* 15:58 bstorm: depooled clouddb1020 for repair [[phab:T291961|T291961]]
* 12:40 dcaro: Merged change on sssd for bullseye cloud hosts ([[phab:T291585|T291585]])
* 11:30 arturo: [codfw1dev] create floating IP 185.15.57.5 for manila-sharecontroller.cloudinfra-codfw1dev.codfw1dev.wmcloud.org ([[phab:T291257|T291257]])

=== 2021-09-27 ===
* 10:07 arturo: cloudcontrol1004 apparently healthy [[phab:T291446|T291446]]
* 09:25 arturo: rebooting cloudcontrol1004 for [[phab:T291446|T291446]]

=== 2021-09-24 ===
* 13:02 arturo: [codfw1dev] create VM manila-share-controller-01 on cloudinfra-codfw1dev
* 13:00 arturo: [codfw1dev] rebase labs/private.git on cloudinfra-puppetmaster-01, had merge conflict

=== 2021-09-21 ===
* 12:13 arturo: [codfw1dev] trying to create a manila service image ([[phab:T291257|T291257]])
* 11:45 arturo: [codfw1dev] created rabbitmq user ([[phab:T291257|T291257]])
* 11:32 arturo: [codfw1dev] populated manila DB &amp; created service endpoints ([[phab:T291257|T291257]])
* 11:06 arturo: [codfw1dev] give manila user admin role @ manila project ([[phab:T291257|T291257]])
* 11:06 arturo: [codfw1dev] created manila project ([[phab:T291257|T291257]])
* 10:57 arturo: [codfw1dev] created manila user @ labtestwikitech ([[phab:T291257|T291257]])
* 10:49 arturo: [codfw1dev] create manila database on cloudcontrol-dev nodes (galera) [[phab:T291257|T291257]]

=== 2021-09-20 ===
* 23:08 bstorm: ran `echo check &gt; /sys/block/md0/md/sync_action` on cloudcontrol1004 to check raid
* 22:48 andrewbogott: stopped puppet &amp; mariadb on cloudcontrol1004; it was flapping
* 22:44 andrewbogott: sudo touch /tmp/galera.disabled on cloudcontrol1004, the service seems troubled there
* 21:57 andrewbogott: moving cloudvirt1043 into the 'nfs' aggregate for [[phab:T291405|T291405]]

=== 2021-09-17 ===
* 11:35 arturo: [codfw1dev] install manila on cloudcontrol2001-dev ([[phab:T291257|T291257]])

=== 2021-09-16 ===
* 15:56 bstorm: removing downtime for labstore1005 so we'll know if it has another issue [[phab:T290318|T290318]]

=== 2021-09-09 ===
* 22:03 bstorm: restarted the prometheus-mysqld-exporter@s1 service as it was not working [[phab:T290630|T290630]]
* 03:15 bstorm: resetting swap on clouddb1017 [[phab:T290630|T290630]]
* 03:08 andrewbogott: stopping maintain-dbusers on labstore1004 for help diagnosing [[phab:T290630|T290630]]

=== 2021-09-03 ===
* 15:34 bstorm: rebooting labstore1005 to disconnect the drives from labstore1004 [[phab:T290318|T290318]]
* 15:24 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2002 [[phab:T290318|T290318]]
* 15:20 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2001 [[phab:T290318|T290318]]

=== 2021-08-30 ===
* 16:16 wm-bot: Added 1 new OSDs ['cloudcephosd1018.eqiad.wmnet'] - cookbook ran by andrew@buster
* 16:16 wm-bot: Added OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:10 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:07 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:07 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:07 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster

=== 2021-08-27 ===
* 18:57 andrewbogott: raising toolsbeta ram/core/instances quotas so majavah can experiment with bullseye

=== 2021-08-25 ===
* 14:45 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 14:42 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 14:42 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 14:42 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 14:41 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster

=== 2021-08-19 ===
* 17:39 bstorm: restarting glance image backup to try and clear the page

=== 2021-08-18 ===
* 16:21 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:21 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:21 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:17 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:16 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:15 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 14:47 andrewbogott: adding clouvirt1038 to the ceph aggregate, removing from the maintenance aggregate [[phab:T276922|T276922]]

=== 2021-08-17 ===
* 15:11 andrewbogott: rebooting cloudcephosd1008 to force raid rebuild -- [[phab:T287838|T287838]]

=== 2021-08-11 ===
* 13:51 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:48 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:47 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 13:47 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-08-10 ===
* 15:15 andrewbogott: restarting all designate services in eqiad1
* 15:04 andrewbogott: restarting designate-sink in eqiad1; it's complaining about rabbit but I don't want to restart rabbit yet

=== 2021-08-05 ===
* 09:37 dcaro: Taking one osd daemon down ot codfw cluster ([[phab:T288203|T288203]])

=== 2021-08-04 ===
* 19:20 bd808: Running deleteBatch.php on cloudweb2001-dev to remove legacy Heira: pages from labtestwiki

=== 2021-08-03 ===
* 17:40 bstorm: rerunning the glance backup script after failure

=== 2021-07-31 ===
* 00:10 andrewbogott: &quot;systemctl reset-failed cloud-init.service&quot; on all VMs for [[phab:T287309|T287309]]
* 00:08 andrewbogott: &quot;systemctl reset-failed cloud-final.service&quot; on all VMs for [[phab:T287309|T287309]]

=== 2021-07-27 ===
* 21:32 andrewbogott: putting cloudvirt1012 back into service [[phab:T286748|T286748]]
* 20:52 andrewbogott: draining VMs off of cloudvirt1012 so we can replace the battery for [[phab:T286748|T286748]]
* 15:15 andrewbogott: &quot;rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list&quot; cloud-wide

=== 2021-07-23 ===
* 15:22 bstorm: update wikireplicas-dns for s7 fix for web replicas

=== 2021-07-20 ===
* 17:07 andrewbogott: reloading haproxy on dbproxy1018 for [[phab:T286598|T286598]]
* 15:45 arturo: failback from labstore1006 to labstore1007 (dumps NFS) https://gerrit.wikimedia.org/r/c/operations/puppet/+/705417
* 00:10 bstorm: restarting nova-api on cloudcontrol1003 to try and recover whatever it's doing with designate_floating_ip_ptr_records_updater

=== 2021-07-19 ===
* 22:05 bstorm: set downtime scheduled for tomorrow from 1300 to 1600 UTC for cloudstore1008 and 1009 [[phab:T286599|T286599]]
* 20:40 andrewbogott: reloading haproxy on dbproxy1018 for [[phab:T286598|T286598]]
* 13:50 andrewbogott: upgrading mariadb to 10.3.29 on all cloudcontrols

=== 2021-07-16 ===
* 09:55 dcaro: checking HP raid issues on coludvirt1012 ([[phab:T286766|T286766]])

=== 2021-07-14 ===
* 21:08 andrewbogott: restarting lots of openstack services while trying to resolve [[phab:T286675|T286675]]
* 12:17 dcaro: doing ceph outage tests on codfw1 (fyi)

=== 2021-07-13 ===
* 10:57 dcaro: enabled autoscaling on codfw1 ceph cluster, setting a minimum of pgs on codfw1dev-compute to 128

=== 2021-07-02 ===
* 10:12 wm-bot: The cluster is not rebalance after adding the new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:12 wm-bot: Added 2 new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:12 wm-bot: Added OSD cloudcephosd1020.eqiad.wmnet... (2/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:10 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding OSD cloudcephosd1020.eqiad.wmnet... (2/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Added OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:02 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:02 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:01 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-07-01 ===
* 16:27 bstorm: failed over cloudstore1009 to cloudstore1008 [[phab:T224747|T224747]]
* 16:18 bstorm: downtimed cloudstore1008 and cloudstore1009 to fail over [[phab:T224747|T224747]]
* 14:25 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (2/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot: Added OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:20 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:20 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Added new OSD node cloudcephosd1016.eqiad.wmnet ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 11:13 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:58 dcaro: rebooting cloudcephosd1016 ([[phab:T285858|T285858]])
* 10:47 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:41 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-30 ===
* 21:48 bstorm: downtimed space alerts for scratch on cloudstore1008 until after the migration

=== 2021-06-25 ===
* 15:28 andrewbogott: restarting openstack services on cloudcontrol1005
* 09:16 arturo: icinga downtime cloudcontrols for 2h
* 08:20 dcaro: restarting rabbitmq on cloudcontrol100&lt;nowiki&gt;{&lt;/nowiki&gt;3,4&lt;nowiki&gt;}&lt;/nowiki&gt;

=== 2021-06-21 ===
* 13:54 dcaro: puppet fix merged and deployed, servers are back to normal
* 13:20 dcaro: merged broken puppet patch, downtimed all cloudvirts for 2h while fixing (nothing big, just added a bad systemd timer)

=== 2021-06-20 ===
* 22:21 andrewbogott: clearing admin-monitoring VMs; puppet has been failing lately due to a full drive on the puppetmaster

=== 2021-06-15 ===
* 01:18 bstorm: running a modified version of the prometheus dir size cron in screen [[phab:T284964|T284964]]

=== 2021-06-14 ===
* 10:13 dcaro: setting ssd to debug mode on tools-sgeexec-0917 ([[phab:T284130|T284130]])

=== 2021-06-10 ===
* 10:58 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:58 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:55 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:55 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:52 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:52 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:49 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:49 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:48 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:48 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:45 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:45 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:39 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:38 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:35 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:35 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:32 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:32 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:29 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:26 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:26 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:24 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:24 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-09 ===
* 17:33 arturo: removed icinga downtime for cloudmetrics1002 -- to see if hardware is healthy ([[phab:T281881|T281881]])
* 13:30 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:30 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-08 ===
* 23:19 bd808: Downtimed cloudmetrics1002 in icinga until 2021-06-30 23:59:01 ([[phab:T281881|T281881]])
* 21:08 bstorm: downtiming grafana-labs for maintenance
* 16:28 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:27 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:24 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:24 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:22 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:21 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:18 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:17 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

=== 2021-06-07 ===
* 14:27 andrewbogott: moving cloudvirt1040 from 'maintenance' aggregate to 'ceph' aggregate  [[phab:T281399|T281399]]

=== 2021-06-01 ===
* 13:12 dcaro: Changed the ceph osd_memory_target on eqiad pool to 6Gi (we were reaching the limit, swapping at some points)
* 09:57 arturo: fix PTR record for 185.15.56.1 ([[phab:T284025|T284025]])
* 09:56 arturo: fix PTR record for 185.15.56.1 ([[phab:T248025|T248025]])

=== 2021-05-27 ===
* 14:58 wm-bot: Testing - cookbook ran by dcaro@vulcanus

=== 2021-05-26 ===
* 19:10 andrewbogott: reimaging cloudvirt1018 to support local VM storage
* 18:07 andrewbogott: draining cloudvirt1018, converting it to a local-storage  host like cloudvirt1019 and 1020 -- [[phab:T283296|T283296]]
* 14:36 dcaro: Enabled syslog logging for osd.55 on eqiad ceph cluster for testing ([[phab:T281247|T281247]])
* 14:36 dcaro: Enabled syslog logging on codfw ceph cluster (mon/osd/mgr) ([[phab:T281247|T281247]])
* 11:26 arturo: [codfw1dev] purge old kernel packages in cloudvirt200[12]-dev
* 11:03 arturo: created public flavor `g3.cores16.ram36.disk20` (even though it was requested as private in [[phab:T283293|T283293]], but may be useful for others)

=== 2021-05-25 ===
* 16:14 bd808: Closed #wikimedia-cloud-admin on f***node
* 16:11 bd808: Closed #wikimedia-cloud-feed on f***node
* 15:19 dcaro: rebooted cloudvirt1020, starting VMs ([[phab:T275893|T275893]])
* 15:13 dcaro: rebooting cloudvirt1020 ([[phab:T275893|T275893]])
* 14:42 dcaro: taking cloudvirt1020 out for maintenance (openstack wise) so no new VMs are scheduled on it ([[phab:T275893|T275893]])

=== 2021-05-24 ===
* 22:32 andrewbogott: changing the default ttl for eqiad1.wikimedia.cloud. from 3600 to 60; this should help us avoid madness when re-using hostnames.
* 11:20 arturo: created `g3.cores2.ram80.disk40.private` for the wmf-research-tools project, to allow resizing a 40G disk instance

=== 2021-05-22 ===
* 02:14 bstorm: downtiming SMART alerts on dumps server labstore1007 for the weekend because it has been flapping [[phab:T281045|T281045]]

=== 2021-05-13 ===
* 21:25 bstorm: converted the maps and scratch volumes on cloudstore1008 (standby) to drbd [[phab:T224747|T224747]]
* 15:45 bstorm: re-running wikireplicas-dns after refactor of config to make sure it doesn't change anything

=== 2021-05-12 ===
* 14:23 arturo: [codfw1dev] cleanup old unused agents (bgp, ovs)
* 11:37 arturo: [codfw1dev] replacing cloudnet2003-dev with cloudnet2004-dev ([[phab:T281381|T281381]])

=== 2021-05-11 ===
* 18:00 andrewbogott: adding 'trove' service project in advance of deploying trove in eqiad1
* 10:22 arturo: rebooted cloudgw1002 (active) thus causing a failover to cloudgw1001

=== 2021-05-09 ===
* 10:53 arturo: icinga-downtime cloudmetrics1002 for 3 months ([[phab:T275605|T275605]])

=== 2021-05-07 ===
* 13:51 andrewbogott: add inherited 'admin' right to novaadmin user throughout eqiad1.  I was trying to narrow down the rights here but lack of admin breaks some workflows, e.g. [[phab:T281894|T281894]] and [[phab:T282235|T282235]]

=== 2021-05-06 ===
* 15:31 arturo: about to migrating CloudVPS network to the cloudgw architecture [[phab:T270704|T270704]]
* 11:14 dcaro: restarting cinder-volume on the eqiad control nodes to refresh the ceph libraries ([[phab:T282109|T282109]])

=== 2021-05-05 ===
* 16:07 dcaro: disallowing insecure global ids on the eqiad ceph cluster ([[phab:T280641|T280641]])
* 15:15 wm-bot: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:11 wm-bot: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:11 wm-bot: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:40 wm-bot: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:39 wm-bot: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:10 wm-bot: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 12:35 wm-bot: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:47 wm-bot: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:47 wm-bot: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:21 wm-bot: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:21 wm-bot: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:45 wm-bot: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:45 wm-bot: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:02 wm-bot: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus

=== 2021-05-04 ===
* 16:05 wm-bot: Safe reboot of 'cloudvirt1028.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:45 wm-bot: Safe rebooting 'cloudvirt1028.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:44 wm-bot: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:22 wm-bot: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:19 wm-bot: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:15 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 13:19 dcaro: rebooting cloudmetrics1002, got stuck again ([[phab:T275605|T275605]])
* 10:04 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:10 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:10 wm-bot: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:34 wm-bot: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:20 wm-bot: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:03 wm-bot: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus

=== 2021-05-03 ===
* 23:53 bstorm: running `maintain-dbusers harvest-replicas` on labstore1004 [[phab:T281287|T281287]]
* 23:51 bstorm: running `maintain-dbusers harvest-replicas` on labstore1004
* 16:34 wm-bot: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 16:29 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:41 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:41 wm-bot: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:31 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T280641|T280641]] - cookbook ran by dcaro@vulcanus)
* 10:23 wm-bot: (from a cookbook)
* 09:12 dcaro: draining and rebooting coludvirt1021 ([[phab:T280641|T280641]])
* 08:26 dcaro: draining and rebooting coludvirt1018 ([[phab:T280641|T280641]])

=== 2021-04-30 ===
* 11:16 dcaro: draining and rebooting coludvirt1017, last one today ([[phab:T280641|T280641]])
* 10:37 dcaro: draining coludvirt1016 for reboot ([[phab:T280641|T280641]])
* 09:48 dcaro: draining coludvirt1013 for reboot ([[phab:T280641|T280641]])

=== 2021-04-29 ===
* 15:11 dcaro: hard rebooting cloudmetrics1002, got hung again ([[phab:T275605|T275605]])
* 07:53 dcaro: Upgrading ceph libraries on cloudcontrol1005 to octopus ([[phab:T274566|T274566]])
* 07:51 dcaro: Upgrading ceph libraries on cloudcontrol1003 to octopus ([[phab:T274566|T274566]])
* 07:50 dcaro: Upgrading ceph libraries on cloudcontrol1004 to octopus ([[phab:T274566|T274566]])

=== 2021-04-28 ===
* 21:11 andrewbogott: cleaning up more references to deleted hypervisors with delete from services where topic='compute' and version != 53;
* 20:48 andrewbogott: cleaning up references to deleted hypervisors with mysql:root@localhost [nova_eqiad1]&gt; delete from compute_nodes where hypervisor_version != '5002000';
* 19:40 andrewbogott: putting cloudvirt1040 into the maintenance aggregate pending more info about [[phab:T281399|T281399]]
* 18:11 andrewbogott: adding cloudvirt1040, 1041 and 1042 to the 'ceph' host aggregate -- [[phab:T275081|T275081]]
* 11:06 dcaro: All ceph server side upgraded to Octopus! \o/ ([[phab:T280641|T280641]])
* 10:57 dcaro: Got a PG getting stuck on 'remapping' after the OSD came up, had to unset the norebalance and then set it again to get it unstuck ([[phab:T280641|T280641]])
* 10:34 dcaro: Slow/blocked opns from cloudcephmon03, &quot;osd_failure(failed timeout osd.32...&quot; (cloudcephosd1005), unset the cluster noout/norebalance and went away in a few secs, setting it again and continuing... ([[phab:T280641|T280641]])
* 09:03 dcaro: Waiting for slow heartbeats from osd.58(cloudcephosd1002) to recover... ([[phab:T280641|T280641]])
* 08:59 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) all from osd.58, currently on cloudcephosd1002 ([[phab:T280641|T280641]])
* 08:58 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) all from osd.58 ([[phab:T280641|T280641]])
* 08:58 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) ([[phab:T280641|T280641]])
* 08:21 dcaro: Upgrading all the ceph osds on eqiad ([[phab:T280641|T280641]])
* 08:21 dcaro: The clock skew seems intermittent, there's another task to follw it [[phab:T275860|T275860]] ([[phab:T280641|T280641]])
* 08:18 dcaro: All equiad ceph mons and mgrs upgraded ([[phab:T280641|T280641]])
* 08:18 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, cloudcephmon1001, they are back ([[phab:T280641|T280641]])
* 08:15 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, it went away, I'm guessing systemd-timesyncd fixed it ([[phab:T280641|T280641]])
* 08:14 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, looking ([[phab:T280641|T280641]])
* 07:58 dcaro: Upgrading ceph services on eqiad, starting with mons/managers ([[phab:T280641|T280641]])

=== 2021-04-27 ===
* 14:10 dcaro: codfw.openstack upgraded ceph libraries to 15.2.11 ([[phab:T280641|T280641]])
* 13:07 dcaro: codfw.openstack cloudvirt2002-dev done, taking cloudvirt2003-dev out to upgrade ceph libraries ([[phab:T280641|T280641]])
* 13:00 dcaro: codfw.openstack cloudvirt2001-dev back online, taking cloudvirt2002-dev out to upgrade ceph libraries ([[phab:T280641|T280641]])
* 10:51 dcaro: ceph.eqiad: cinder pool got it's pg_num increased to 1024, re-shuffle started ([[phab:T273783|T273783]])
* 10:48 dcaro: ceph.eqiad: Tweaked the target_size_ratio of all the pools, enabling autoscaler (it will increase cinder pool only) ([[phab:T273783|T273783]])
* 09:14 dcaro: manually force stopping the server puppetmaster-01 to unblock migration (in codfw1)
* 09:14 dcaro: manually force stopping the server puppetmaster-01 to unblock migration
* 08:59 dcaro: manually force stopping the server exploding-head on codfw, to try cold migration
* 08:47 dcaro: restarting nova-compute on cloudvirt2001-dev after upgrading ceph libraries to 15.2.11

=== 2021-04-26 ===
* 20:56 andrewbogott: deleting spurious 'codfw1dev' and 'codw1dev-4' regions in the dallas deployment; regions without endpoints break a bunch of things
* 09:45 dcaro: draining cloudvirt2001-dev with the new cookbooks ([[phab:T280641|T280641]])

=== 2021-04-23 ===
* 13:49 dcaro: testing the drain_cloudvirt cookbook on codfw1 openstack cluster, draining cloudvirt2001 ([[phab:T280641|T280641]])
* 11:12 dcaro: testing the drain_cloudvirt cookbook on codfw1 openstack cluster ([[phab:T280641|T280641]])
* 09:32 dcaro: finished upgrade of ceph cluster on codfw1 using exclusively cookbooks ([[phab:T280641|T280641]])
* 09:17 dcaro: testing the upgrade_osds cookbook on codfw1 ceph cluster ([[phab:T280641|T280641]])
* 08:17 dcaro: testing the upgrade_mons cookbook on codfw1 ceph cluster ([[phab:T280641|T280641]])

=== 2021-04-21 ===
* 17:59 dcaro: all monitors upgraded on codfw1 with one cookbook `cookbook --verbose -c ~/.config/spicerack/cookbook.yaml wmcs.ceph.upgrade_mons --monitor-node-fqdn cloudcephmon2002-dev.codfw.wmnet` ([[phab:T280641|T280641]])
* 17:47 dcaro: upgrading monitors and mrg nodes on codfw ceph cluster ([[phab:T280641|T280641]])
* 13:26 dcaro: testing ceph upgrade cookbook on cloudcephmon2002-dev ([[phab:T280641|T280641]])

=== 2021-04-20 ===
* 20:21 andrewbogott: reboot cloudservices1003
* 20:13 andrewbogott: reboot cloudservices1004

=== 2021-04-19 ===
* 08:40 dcaro: enabling puppet on labstore1004 after mysql restart ([[phab:T279657|T279657]])
* 08:09 dcaro: downtiming labstore1004 and stopping puppet for mysql restart ([[phab:T279657|T279657]])

=== 2021-04-14 ===
* 10:48 dcaro: Upgrade of codfw ceph to octopus 15.2.20 done, will run some performance tests now ([[phab:T274566|T274566]])
* 10:41 dcaro: Upgrade of codfw ceph to octopus 15.2.20, mgrs upgraded, osds next ([[phab:T274566|T274566]])
* 10:37 dcaro: Upgrade of codfw ceph to octopus 15.2.20, mons upgraded, mgrs next ([[phab:T274566|T274566]])
* 10:15 dcaro: starting the upgrade of codfw ceph to octopus 15.2.20 ([[phab:T274566|T274566]])
* 10:07 dcaro: Merged the ceph 15 (Octopus) repo deployment to codfw, only the repo, not the packages ([[phab:T274566|T274566]])

=== 2021-04-13 ===
* 16:42 dcaro: Ceph balancer got the cluster to eval 0.014916, that is 88-77% usage for compute pool, and 28-19% usage for the cinder one \o/ ([[phab:T274573|T274573]])
* 15:08 dcaro: Activating continuous upmap balancer, keeping a close eye ([[phab:T274573|T274573]])
* 15:03 dcaro: Executing a second pass, there's still movements to improve the eval of 0.030075 ([[phab:T274573|T274573]])
* 15:02 dcaro: First pass finished, improved eval to 0.030075 ([[phab:T274573|T274573]])
* 14:49 dcaro: Running the first_pass balancing plan on ceph eqiad, current eval 0.030622 ([[phab:T274573|T274573]])
* 14:43 dcaro: enabling ceph upmap pg balancer on equiad ([[phab:T274573|T274573]])
* 14:36 andrewbogott: upgrading codfw1dev to version Victoria, [[phab:T261137|T261137]]
* 13:11 andrewbogott: upgrading eqiad1 designate to version Victoria, [[phab:T261137|T261137]]
* 10:44 dcaro: enabled ceph upmap balancer on codfw ([[phab:T274573|T274573]],[[phab:T274573|T274573]])

=== 2021-04-07 ===
* 21:33 andrewbogott: upgrading codfw1dev designate to Victoria

=== 2021-04-04 ===
* 17:36 andrewbogott: upgrading eqiad1 designate to Ussuri

=== 2021-04-02 ===
* 14:12 andrewbogott: upgrading codfw1dev to OpenStack version Ussuri

=== 2021-04-01 ===
* 12:15 dcaro: Restoring the 4.9 kernel on cloudcephosd2003-dev and upgrading ([[phab:T274565|T274565]])
* 10:29 dcaro: Done restoring the 4.9 kernel on cloudcephosd2001-dev and upgrading, requires logging into console to boot from the older kernel before removing the newer one ([[phab:T274565|T274565]])
* 10:10 dcaro: Restoring the 4.9 kernel on cloudcephosd2001-dev and upgrading ([[phab:T274565|T274565]])

=== 2021-03-31 ===
* 08:47 dcaro: upgrading cinder on codfw cloudcontrol2* nodes ([[phab:T278845|T278845]])

=== 2021-03-30 ===
* 09:53 arturo: rebooting cloudnet1003 to cleanup conntrack table, it wouldn't cleanup by hand ...

=== 2021-03-28 ===
* 15:42 andrewbogott: updated debian-10.0-buster base image

=== 2021-03-27 ===
* 09:54 arturo: cleanup conntrack table in qrouter nents in cloudnet1003 (backup)

=== 2021-03-25 ===
* 19:03 andrewbogott: deleting all unused (per wmcs-imageusage) Jessie base images from Glance
* 17:15 andrewbogott: refreshing puppet compiler facts for tools project
* 10:31 dcaro: kernel upgrade on osds on codfw done, running performance tests ([[phab:T274565|T274565]])
* 10:24 dcaro: upgrading kernel on cloudcephosd2003-dev and reboot ([[phab:T274565|T274565]])
* 10:18 dcaro: upgrading kernel on cloudcephosd2002-dev and reboot ([[phab:T274565|T274565]])
* 10:08 dcaro: upgrading kernel on cloudcephmon2003-dev and reboot ([[phab:T274565|T274565]])

=== 2021-03-24 ===
* 09:19 dcaro: restarted wmcs-backup on cloudvirt1024 as it failed due to an image being removed while running ([[phab:T276892|T276892]])

=== 2021-03-23 ===
* 11:33 arturo: root@cloudcontrol1005:~# wmcs-novastats-dnsleaks --delete

=== 2021-03-22 ===
* 10:10 arturo: cleanup conntrack table in standby node: aborrero@cloudnet1003:~ $ sudo ip netns exec qrouter-d93771ba-2711-4f88-804a-{{Gerrit|8df6fd03978a}} conntrack -F

=== 2021-03-19 ===
* 17:18 bstorm: running `ALTER TABLE account MODIFY COLUMN type ENUM('user','tool','paws');` against the labsdbaccounts database on m5 [[phab:T276284|T276284]]
* 14:29 andrewbogott: switching admin-monitoring project to use an upstream debian image; I want to see how this affects performance
* 00:30 bstorm: downtimed labstore1004 to check some things in debug mode

=== 2021-03-17 ===
* 17:28 bstorm: restarted the backup-glance-images job to clear errors in systemd [[phab:T271782|T271782]]
* 17:16 andrewbogott: set default cinder quota for projects to 80Gb with &quot;update quota_classes set hard_limit=80 where resource='gigabytes';&quot; on database 'cinder'
* 16:58 andrewbogott: disabling all flavors with &gt;20Gb root storage with &quot;update flavors set disabled=1 where root_gb&gt;20;&quot; in nova_eqiad1_api

=== 2021-03-10 ===
* 16:51 arturo: rebooting cloudvirt1030 for [[phab:T275753|T275753]]
* 13:14 dcaro: starting manually the canary VM for cloudvirt1029 (nova start 349830f6-3b39-4a8c-ada4-{{Gerrit|a7439f65cffe}}) ([[phab:T275753|T275753]])
* 12:51 arturo: draining cloudvirt1030 for [[phab:T275753|T275753]]
* 12:47 arturo: rebooting cloudvirt1029 for [[phab:T275753|T275753]]
* 11:56 arturo: [codfw1dev] restart rabbitmq-server in all 3 cloudcontrol servers for [[phab:T276964|T276964]]
* 11:53 arturo: [codfw1dev] restart nova-conductor in all 3 cloudcontrol servers for [[phab:T276964|T276964]]
* 11:31 arturo: draining cloudvirt1029 for [[phab:T275753|T275753]]
* 11:29 arturo: rebooting cloudvirt1013 for [[phab:T275753|T275753]]
* 11:05 arturo: draining cloudvirt1013 for [[phab:T275753|T275753]]
* 11:00 arturo: rebooting cloudvirt1028 for [[phab:T275753|T275753]]
* 10:33 arturo: draining cloudvirt1028 for [[phab:T275753|T275753]]
* 10:29 arturo: rebooting cloudvirt1023 for [[phab:T275753|T275753]]
* 09:37 arturo: draining cloudvirt1023 for [[phab:T275753|T275753]]
* 09:07 arturo: [codfw1dev] reimaging cloudvirt2003-dev ([[phab:T276964|T276964]])

=== 2021-03-09 ===
* 16:27 arturo: rebooting cloudvirt1027 ([[phab:T275753|T275753]])
* 13:39 arturo: draining cloudvrit1027 for [[phab:T275753|T275753]]
* 13:35 arturo: icinga-downtime cloudvirt1038 for 30 days for [[phab:T276922|T276922]]
* 13:21 arturo: add cloudvirt1039 to the ceph host aggregate (no longer a spare, we have cloudvirt1038 with HW failures)
* 12:52 arturo: cloudvirt1038 hard powerdown / powerup for [[phab:T276922|T276922]]
* 12:33 arturo: rebooting cloudvirt1038 ([[phab:T275753|T275753]])
* 10:58 arturo: draining cloudvirt1038 ([[phab:T275753|T275753]])
* 10:54 arturo: rebooting cloudvirt1037 ([[phab:T275753|T275753]])
* 09:59 arturo: draining cloudvirt1037 ([[phab:T275753|T275753]])
* 09:12 dcaro: restarted the wmcs-backup service on cloudvirt1024 to retry the backups (failed because a VM was removed in-between, [[phab:T276892|T276892]])

=== 2021-03-05 ===
* 21:40 andrewbogott: replacing 'observer' role with 'reader' role in eqiad1 [[phab:T276018|T276018]]
* 21:21 andrewbogott: replacing 'observer' role with 'reader' role in eqiad1
* 16:23 arturo: rebooting cloudvirt1036 for [[phab:T275753|T275753]]
* 12:30 arturo: draining cloudvirt1036 for [[phab:T275753|T275753]]
* 12:25 arturo: rebooting cloudvirt1035 for [[phab:T275753|T275753]]
* 10:49 arturo: rebooting cloudvirt1035 for [[phab:T275753|T275753]]
* 10:47 arturo: rebooting cloudvirt1034 for [[phab:T275753|T275753]]
* 10:26 arturo: draining cloudvirt1034 for [[phab:T275753|T275753]]
* 10:25 arturo: rebooting cloudvirt1033 for [[phab:T275753|T275753]]
* 09:18 arturo: draining cloudvirt1033 for [[phab:T275753|T275753]]

=== 2021-03-04 ===
* 18:36 andrewbogott: rebooting cloudmetrics1002; the console is hanging
* 16:59 arturo: rebooting cloudvirt1032 for [[phab:T275753|T275753]]
* 16:34 arturo: draining cloudvirt1032 for [[phab:T275753|T275753]]
* 16:33 arturo: rebooting cloudvirt1031 for [[phab:T275753|T275753]]
* 16:11 arturo: draining cloudvirt1031 for [[phab:T275753|T275753]]
* 16:09 arturo: rebooting cloudvirt1026 for [[phab:T275753|T275753]]
* 15:57 arturo: draining cloudvirt1026 for [[phab:T275753|T275753]]
* 15:55 arturo: rebooting cloudvirt1025 for [[phab:T275753|T275753]]
* 15:41 arturo: draining cloudvirt1025 for [[phab:T275753|T275753]]
* 15:12 arturo: rebooting cloudvirt1024 for [[phab:T275753|T275753]]
* 11:29 arturo: draining cloudvirt1024 for [[phab:T275753|T275753]]
* 11:24 dcaro: rebooted cloudvirt1022, re-adding to ceph and removing from maintenance host aggregate for [[phab:T275753|T275753]]
* 11:01 dcaro: rebooting cloudvirt1022 for [[phab:T275753|T275753]]
* 09:12 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]

=== 2021-03-03 ===
* 17:16 andrewbogott: restarting rabbitmq-server on cloudcontrol1003,1004,1005; trying to explain amqp errors in scheduler logs
* 16:03 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]
* 16:03 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]
* 16:00 arturo: move cloudvirt1013 into the 'toobusy' host aggregate, it has 221% cpu subscription and 82% MEM subscription
* 15:34 arturo: rebooting cloudvirt1021 for [[phab:T275753|T275753]]
* 14:31 arturo: draining cloudvirt1021 for [[phab:T275753|T275753]]
* 13:59 arturo: rebooting cloudvirt1018 for [[phab:T275753|T275753]]
* 13:28 arturo: draining cloudvirt1018 for [[phab:T275753|T275753]]
* 12:49 arturo: rebooting cloudvirt1017 for [[phab:T275753|T275753]]
* 12:22 arturo: draining cloudvirt1017 for [[phab:T275753|T275753]]
* 12:20 arturo: rebooting cloudvirt1016 for [[phab:T275753|T275753]]
* 12:01 arturo: draining cloudvirt1016 for [[phab:T275753|T275753]]
* 11:59 arturo: cloudvirt1014 now in the ceph host aggregate
* 11:58 arturo: rebooting cloudvirt1014 for [[phab:T275753|T275753]]
* 11:50 arturo: moved cloudvirt1023 away from the maintenance host aggregate, leave it in the ceph aggregate (was in the 2)
* 11:47 arturo: moved cloudvirt1014 to the 'maintenance' host aggregate, drain it for [[phab:T275753|T275753]]
* 10:01 arturo: icinga-downtime cloudnet1003 for 14 days bc potential alerting storm due to firmware issues ([[phab:T271058|T271058]])
* 10:01 arturo: rebooting again cloudnet1003 (no network failover) ([[phab:T271058|T271058]])
* 09:59 arturo: update firmware-bnx2x from 20190114-2 to 20200918-1~bpo10+1 on cloudnet1003 ([[phab:T271058|T271058]])
* 09:30 arturo: installing linux kernel 5.10.13-1~bpo10+1 in cloudnet1003 and rebooting it (network failover) ([[phab:T271058|T271058]])

=== 2021-03-02 ===
* 17:16 andrewbogott: rebooting cloudvirt1039 to see if I can trigger [[phab:T276208|T276208]]
* 16:10 arturo: [codfw1dev] restart nova-compute on cloudvirt2002-dev
* 11:59 arturo: moved cloudvirt1012 to 'maintenance' host aggregate. Drain it with `wmcs-drain-hypervisor` to reboot it for [[phab:T275753|T275753]]
* 11:59 arturo: cloudvirt1023 is affected by [[phab:T276208|T276208]] and cannot be rebooted. Put it back into the ceph hos aggregate
* 10:43 arturo: moved cloudvirt1013 cloudvirt1032 cloudvirt1037 back into the 'ceph' host aggregate
* 10:13 arturo: moved cloudvirt1023 to 'maintenance' host aggregate. Drain it with `wmcs-drain-hypervisor` to reboot it for [[phab:T275753|T275753]]

=== 2021-03-01 ===
* 20:12 andrewbogott: removing novaadmin from all projects save 'admin' for [[phab:T274385|T274385]]
* 19:51 andrewbogott: removing novaobserver from all projects save 'observer' for [[phab:T274385|T274385]]
* 19:50 andrewbogott: adding inherited domain-wide roles to novaadmin and novaobserver as per [[phab:T274385|T274385]]

=== 2021-02-28 ===
* 04:54 andrewbogott: restarted redis-server on tools-redis-1003 and tools-redis-1004 in an attempt to reduce replag, no real change detected

=== 2021-02-27 ===
* 00:33 andrewbogott: sudo cumin --timeout 500 &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i buster &amp;&amp; uname -r  {{!}} grep -v 4.19.0-14-amd64 &amp;&amp; reboot'
* 00:28 andrewbogott: sudo cumin --timeout 500 &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i buster &amp;&amp; uname -r  {{!}} grep -v 4.19.0-14-amd64 &amp;&amp; echo reboot'
* 00:09 andrewbogott: sudo cumin &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i stretch &amp;&amp; uname -r  {{!}} grep -v 4.19.0-0.bpo.14-amd64 &amp;&amp; reboot'

=== 2021-02-26 ===
* 14:58 dcaro: [eqiad] rebooting cloudcephosd1015 (last osd \o/) for kernel upgrade ([[phab:T275753|T275753]])
* 14:51 dcaro: [eqiad] rebooting cloudcephosd1014 for kernel upgrade ([[phab:T275753|T275753]])
* 14:44 dcaro: [eqiad] rebooting cloudcephosd1013 for kernel upgrade ([[phab:T275753|T275753]])
* 14:38 dcaro: [eqiad] rebooting cloudcephosd1012 for kernel upgrade ([[phab:T275753|T275753]])
* 14:31 dcaro: [eqiad] rebooting cloudcephosd1011 for kernel upgrade ([[phab:T275753|T275753]])
* 14:25 dcaro: [eqiad] rebooting cloudcephosd1010 for kernel upgrade ([[phab:T275753|T275753]])
* 14:17 dcaro: [eqiad] rebooting cloudcephosd1009 for kernel upgrade ([[phab:T275753|T275753]])
* 13:54 dcaro: [eqiad] downtimed alert1001 Ceph OSDs down alert until 18:00 GMT+1 as that is not under the host being rebooted ([[phab:T275753|T275753]])
* 13:51 dcaro: [eqiad] rebooting cloudcephosd1008 for kernel upgrade ([[phab:T275753|T275753]])
* 13:45 dcaro: [eqiad] rebooting cloudcephosd1007 for kernel upgrade ([[phab:T275753|T275753]])
* 13:38 dcaro: [eqiad] rebooting cloudcephosd1006 for kernel upgrade ([[phab:T275753|T275753]])
* 12:07 dcaro: [eqiad] rebooting cloudcephosd1005 for kernel upgrade ([[phab:T275753|T275753]])
* 12:00 arturo: rebooting cloudcontrol1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:42 arturo: rebooting cloudcontrol1004 for kernel upgrade ([[phab:T275753|T275753]])
* 11:41 dcaro: [eqiad] rebooting cloudcephosd1004 for kernel upgrade ([[phab:T275753|T275753]])
* 11:32 dcaro: [eqiad] rebooting cloudcephosd1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:30 arturo: rebooting cloudcontrol1005 for kernel upgrade ([[phab:T2|T2]]
* 11:26 dcaro: [eqiad] rebooting cloudcephosd1002 for kernel upgrade ([[phab:T275753|T275753]])
* 11:16 dcaro: [eqiad] rebooting cloudcephosd1001 for kernel upgrade ([[phab:T275753|T275753]])
* 11:11 dcaro: [eqiad] rebooting cloudcephmon1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:05 dcaro: [eqiad] rebooting cloudcephmon1002 for kernel upgrade ([[phab:T275753|T275753]])
* 10:59 dcaro: [eqiad] rebooting cloudcephmon1001 for kernel upgrade ([[phab:T275753|T275753]])
* 10:45 arturo: rebooting cloudvirt1039 into a new kernel ([[phab:T275753|T275753]]) --- spare
* 10:43 dcaro: [codfw1dev] rebooting cloudcephmon2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:38 dcaro: [codfw1dev] rebooting cloudcephmon2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:29 dcaro: [codfw1dev] rebooting cloudcephmon2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:24 arturo: [codfw1dev] purge old kernel packages on cloudvirt2003-dev to force boot into a new kernel ([[phab:T275753|T275753]])
* 10:11 arturo: [codfw1dev] manually creating /boot/grub/ on cloudvirt2003-dev to allow update-grub2 to run (so it can reboot into a new kernel) ([[phab:T275753|T275753]])
* 10:11 dcaro: [codfw1dev] rebooting cloudcephosd2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:05 dcaro: [codfw1dev] rebooting cloudcephosd2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:01 arturo: [codfw1dev] rebooting cloudvirt200X-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:59 arturo: [codfw1dev] rebooting cloudweb2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:53 arturo: [codfw1dev] rebooting cloudservices2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:51 arturo: [codfw1dev] rebooting cloudservices2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:45 arturo: [codfw1dev] rebooting cloudcontrol2004-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:44 arturo: [codfw1dev] rebooting cloudbackup[2001-2002].codfw.wmnet for kernel upgrade ([[phab:T275753|T275753]])
* 09:43 dcaro: [codfw1dev] rebooting cloudcephosd2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:41 arturo: [codfw1dev] rebooting cloudcontrol2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:33 arturo: [codfw1dev] rebooting cloudcontrol2001-dev for kernel upgrade ([[phab:T275753|T275753]])

=== 2021-02-25 ===
* 14:56 arturo: deployed wmcs-netns-events daemon to all cloudnet servers ([[phab:T275483|T275483]])

=== 2021-02-24 ===
* 11:07 arturo: force-reboot cloudmetrics1002, add icinga downtime for 2 hours. Investigating some server issue
* 00:17 bstorm: set  --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main stretch image in glance on eqiad1 [[phab:T275430|T275430]]

=== 2021-02-23 ===
* 22:43 bstorm: set  --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main buster image in glance on eqiad1 [[phab:T275430|T275430]]
* 20:36 andrewbogott: adding r/o access to the eqiad1-glance-images ceph pool for the client.eqiad1-compute for [[phab:T275430|T275430]]
* 10:49 arturo: rebooting clounet1004 into new kernel from buster-bpo ([[phab:T271058|T271058]])
* 10:49 arturo: installing linux-image-amd64 from buster-bpo 5.10.13-1~bpo10+1 in cloudnet1004 ([[phab:T271058|T271058]])

=== 2021-02-22 ===
* 17:15 bstorm: restarting nova-compute on cloudvirt1016 and cloudvirt1036 in case it helps [[phab:T275411|T275411]]
* 15:02 dcaro: Re-uploaded the debian buster 10.0 image from rbd to glance, that worked, re-spawning all the broken instances ([[phab:T275378|T275378]])
* 11:12 dcaro: Refreshing all the canary instances ([[phab:T275354|T275354]])

=== 2021-02-18 ===
* 14:50 arturo: rebooting cloudnet1004 for [[phab:T271058|T271058]]
* 10:25 dcaro: Rebooting cloudmetrics1001 to apply new kernel ([[phab:T275116|T275116]])
* 10:16 dcaro: Rebooting cloudmetrics1002 to apply new kernel ([[phab:T275116|T275116]])
* 10:14 dcaro: Upgrading grafana on cloudmetrics1002 ([[phab:T275116|T275116]])
* 10:12 dcaro: Upgrading grafana on cloudmetrics1001 ([[phab:T275116|T275116]])

=== 2021-02-17 ===
* 15:58 arturo: deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/664845 to cloudnet servers ([[phab:T268335|T268335]])

=== 2021-02-15 ===
* 16:25 arturo: [codfw1dev] rebooting all cloudgw200x-dev / cloudnet200x-dev servers ([[phab:T272963|T272963]])
* 15:45 arturo: [codfw1dev] drop subnet definition for cloud-instances-transport1-b-codfw ([[phab:T272963|T272963]])
* 15:45 arturo: [codfw1dev] connect virtual router cloudinstances2b-gw to vlan cloud-gw-transport-codfw (185.15.57.10) ([[phab:T272963|T272963]])

=== 2021-02-11 ===
* 12:01 arturo: [codfw1dev] drop instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` (was buster, cannot use it yet)
* 11:59 arturo: [codfw1dev] create instance `tools-codfw1dev-bastion-2` (stretch) in `tools-codfw1dev` to test stuff related to [[phab:T272397|T272397]]
* 11:45 arturo: [codfw1dev] create instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` to test stuff related to [[phab:T272397|T272397]]
* 11:42 arturo: [codfw1dev] drop `tools` project, create `tools-codfw1dev`
* 11:38 arturo: [codfw1dev] drop `coudinfra` project (we are using `cloudinfra-codfw1dev` there)
* 05:37 bstorm: downtimed cloudnet1004 for another week [[phab:T271058|T271058]]

=== 2021-02-09 ===
* 15:23 arturo: icinga-downtime for 2h everything *labs *cloud for openstack upgrades
* 11:14 dcaro: Merged the osd scheduler change for all osds, applying on all cloudcephosd* ([[phab:T273791|T273791]])

=== 2021-02-08 ===
* 18:50 bstorm: enabled puppet on cloudvirt1023 for now [[phab:T274144|T274144]]
* 18:44 bstorm: restarted the backup_vms.service on cloudvirt1027 [[phab:T274144|T274144]]
* 17:51 bstorm: deleted project pki [[phab:T273175|T273175]]

=== 2021-02-05 ===
* 10:59 arturo: icinga-downtime labstore1004 tools share space check for 1 week ([[phab:T272247|T272247]])
* 10:21 dcaro: This was affecting maps and several others, maps and project-proxy have been fixed ([[phab:T273956|T273956]])
* 09:19 dcaro: Some certs around the infra are expired ([[phab:T273956|T273956]])

=== 2021-02-04 ===
* 10:12 dcaro: Increasing the memory limit of osds in eqiad from 8589934592(8G) to 12884901888(12G) ([[phab:T273851|T273851]])

=== 2021-02-03 ===
* 09:59 dcaro: Doing a full vm backup on cloudvirt1024 with the new script ([[phab:T260692|T260692]])
* 01:50 bstorm: icinga-downtime cloudnet1004 for a week [[phab:T271058|T271058]]

=== 2021-02-02 ===
* 17:14 dcaro: Changed osd memory limit from 4G to 8G ([[phab:T273649|T273649]])
* 11:00 arturo: icinga-downtime cloudvirt-wdqs1001 for 1 week ([[phab:T273579|T273579]])
* 03:12 andrewbogott: running /usr/local/sbin/wmcs-purge-backups and /usr/local/sbin/wmcs-backup-instances on cloudvirt1024 to see why the backup job paged

=== 2021-01-29 ===
* 15:36 andrewbogott: disabling puppet and some services on eqiad1 cloudcontrol nodes; replacing nova-placement-api with placement-api

=== 2021-01-28 ===
* 19:44 andrewbogott: shutting down cloudcontrol2001-dev because it's in a partially upgraded state; will revive when it's time for Train

=== 2021-01-27 ===
* 00:50 bstorm: icinga-downtime cloudnet1004 for a week [[phab:T271058|T271058]]

=== 2021-01-22 ===
* 16:44 andrewbogott: upgrading designate on cloudvirt1003/1004 to OpenStack 'train'
* 11:29 dcaro: Doing some tests removed cloudcontrol1003 puppet cert, regenerating...

=== 2021-01-21 ===
* 11:35 arturo: merging core router firewall changes https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657439 ([[phab:T209082|T209082]])
* 11:30 arturo: merging core router firewall changes https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657358 ([[phab:T272486|T272486]], [[phab:T209082|T209082]])

=== 2021-01-20 ===
* 10:49 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657302 ([[phab:T209082|T209082]])
* 10:05 dcaro: Everything looks ok, created a new vm with a volume in ceph without issues, and on warnings/errors on ceph status, closing ([[phab:T272303|T272303]])
* 09:55 dcaro: Eqiad ceph cluster uprgaded, doing sanity checks ([[phab:T272303|T272303]])
* 09:46 dcaro: 75% of the eqiad cluster upgraded... continuing ([[phab:T272303|T272303]])
* 09:37 dcaro: 25% of the eqiad cluster upgraded... continuing ([[phab:T272303|T272303]])
* 09:24 dcaro: Mgr daemons upgraded and running, upgrading osd daemons on servers cloudcephosd1*, this make take a bit longer ([[phab:T272303|T272303]])
* 09:22 dcaro: Mon daemons upgraded and running, upgrading mgr daemons on servers cloudcephmon1* ([[phab:T272303|T272303]])
* 09:16 dcaro: Starting eqiad ceph upgrade, upgrading the mon servers cloudcephmon1* ([[phab:T272303|T272303]])
* 09:01 dcaro: Will start the ceph upgrade in 15 min, no downtime nor performance impact is expected ([[phab:T272303|T272303]])

=== 2021-01-19 ===
* 10:17 arturo: icinga-downtime cloudnet1004 for 1 week ([[phab:T271058|T271058]])

=== 2021-01-18 ===
* 16:00 dcaro: Codfw1 ceph cluster uprgaded, will wait until tomorrow to see if there's any instability, but everything looks fine ([[phab:T272303|T272303]])
* 15:38 dcaro: Upgraded mgr sevices on codfw ceph cluster, starting with osd ones ([[phab:T272303|T272303]])
* 15:35 dcaro: Upgraded mon sevices on codfw ceph cluster, starting with mgr ones ([[phab:T272303|T272303]])
* 15:21 dcaro: Starting upgrade of ceph mon nodes on codfw ([[phab:T272303|T272303]])
* 15:06 dcaro: re-enabling puppet on cloudcephosd2* hosts
* 13:53 dcaro: disabling puppet on cloudcephosd2* to resume perf tests
* 10:50 dcaro: re-enabling puppet on cephcloudosd2* (codfw)
* 10:07 dcaro: disabling puppet on cephcloudosd2* (codfw) to do some performance tests
* 09:00 dcaro: Enabling custom application 'cinder' on pool codfw1dev-cinder to get rid of health warnings

=== 2021-01-17 ===
* 16:53 arturo: icinga downtime labstore1004 /srv/tools space check for 3 days ([[phab:T272247|T272247]])

=== 2021-01-15 ===
* 13:41 arturo: icinga downtime labstore1004 maintain-dbuser alert until 2021-01-19 ([[phab:T272125|T272125]])
* 09:47 arturo: labstore1004 maintain-dbusers affected by [[phab:T272127|T272127]] and [[phab:T272125|T272125]]
* 09:22 arturo: restart maintain-dbusers.service in labstore1004
* 08:19 dcaro: Merging the patch to disable write caches on ceph osds ([[phab:T271527|T271527]])

=== 2021-01-13 ===
* 17:03 arturo: remove cloudvirt1013 cloudvirt1032 cloudvirt1037 to the 'toobusy' host aggregate to prevent further CPU oversubscribing
* 12:40 arturo: try increasing systemd watchdog timeout for conntrackd in cloudnet1004 ([[phab:T268335|T268335]])
* 11:45 dcaro: https://gerrit.wikimedia.org/r/c/operations/puppet/+/654419 merged and deployed (and tested) ([[phab:T268877|T268877]])
* 11:40 dcaro: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/654419 that might affect the encapi service (puppet on cloud environment), no downtime expected though ([[phab:T268877|T268877]])
* 10:56 arturo: trying to cleanup dpkg package mess in cloudnet2002-dev
* 10:02 arturo: prevent floating IP allocation from neutron transport subnet: root@cloudcontrol1005:~# neutron subnet-update --allocation-pool start=185.15.56.244,end=185.15.56.244 cloud-instances-transport1-b-eqiad1 ([[phab:T271867|T271867]])

=== 2021-01-12 ===
* 10:33 arturo: reboot cloudnet1004
* 10:32 arturo: update firmware-bnx2x from 20190114-2 to 20200918-1~bpo10+1 on cloudnet1004 ([[phab:T271058|T271058]])

=== 2021-01-11 ===
* 10:22 arturo: doubling size of conntrack table in cloudnet servers https://gerrit.wikimedia.org/r/c/operations/puppet/+/655407 ([[phab:T271058|T271058]])
* 10:07 arturo: manually cleanup conntrack table in cloudnet1004 ([[phab:T271058|T271058]])
* 09:19 dcaro: cleaned up ~1800 snapshots, 109 remaining only, one for each host x image combination (plus some ephemeral ones while doing backups), closing the task ([[phab:T270478|T270478]])
* 08:39 dcaro: cleaning up dangling snapshots now that we have the new suffixed ones ([[phab:T270478|T270478]])

=== 2021-01-10 ===
* 16:02 andrewbogott: restarting rabbitmq-server on all eqiad1 cloudcontrols
* 15:54 andrewbogott: restating neutron-metadata-agent on cloudnet1004 due to many syslog complaints

=== 2021-01-08 ===
* 11:25 arturo: rebooting both cloudnet2002-dev/cloudnet2003-dev to make sure interfaces are set up correctl ([[phab:T271517|T271517]])
* 11:22 arturo: connecting cloudnet2002-dev cloudnet2003-dev back to vlan 2120 ([[phab:T271517|T271517]])
* 11:06 arturo: root@cloudcontrol2001-dev:~# openstack router set --external-gateway wan-transport-codfw --fixed-ip subnet=cloud-instances-transport1-b-codfw,ip-address=208.80.153.190 cloudinstances2b-gw ([[phab:T271517|T271517]])
* 11:02 arturo: root@cloudcontrol2001-dev:~# openstack router set --enable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw ([[phab:T271517|T271517]])
* 11:01 arturo: enabling neutron hacks in codfw1dev (cloudnet2002-dev, cloudnet2003-dev) ([[phab:T271517|T271517]])
* 10:55 arturo: aborrero@labtestvirt2003:~ $ sudo ifdown eno2.2107 ([[phab:T271517|T271517]])
* 10:55 arturo: aborrero@labtestvirt2003:~ $ sudo ifdown eno2.2120 ([[phab:T271517|T271517]])
* 10:53 arturo: root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 208.80.153.185 --ip-version 4 --network wan-transport-codfw --no-dhcp --subnet-range 208.80.153.184/29 cloud-instances-transport1-b-codfw ([[phab:T271517|T271517]])
* 10:40 dcaro: Finished tests, brining osd online (od.48) for eqiad ceph cluster ([[phab:T271417|T271417]])
* 09:59 dcaro: Started performance tests on sdc (od.48) for eqiad ceph cluster ([[phab:T271417|T271417]])
* 09:41 dcaro: Taking osd.48 from eqiad ceph cluster out to do performance tests ([[phab:T271417|T271417]])

=== 2021-01-07 ===
* 15:19 dcaro: Finished speed tests on cloudcephosd2001-dev, reprovisioning the osd.0 sdc ([[phab:T271417|T271417]])
* 14:39 dcaro: Starting speed tests on cloudcephosd2001-dev sdc ([[phab:T271417|T271417]])
* 12:54 dcaro: Taking osd.0 down on codfw ceph cluster to try the disk performance testing process ([[phab:T271417|T271417]])
* 11:35 arturo: merging dmz_cidr change ([[phab:T209082|T209082]], [[phab:T267779|T267779]])

=== 2021-01-05 ===
* 10:40 dcaro: removing dumps-[1..*] backups from cloudvirt1024 as they are not needed ([[phab:T271094|T271094]])

=== 2021-01-03 ===
* 07:06 dcaro: Got a network hiccup on cloudnet1004, keeping track here [[phab:T271058|T271058]]

=== 2020-12-28 ===
* 12:32 arturo: stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 ([[phab:T260692|T260692]])
* 12:32 arturo: stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 ([[phab:T260682|T260682]])
* 12:23 arturo: icinga downtime cloudvirt1026 disk space check until january 5 ([[phab:T260692|T260692]])
* 06:15 andrewbogott: restarting designate-central on cloudservices1003/1004.  I'm pretty sure they're distressed because of DB lag but it's worth a try

=== 2020-12-23 ===
* 15:38 andrewbogott: restarting rabbitmq on cloudcontrol1004; suspected leaks
* 15:33 andrewbogott: restarting each cloudcontrol galera node in turn to see if that quiets down the syncing warnings
* 12:08 arturo: move memory out of the swap in cloudcontrol1004 by disabling/enabling it (1Gb swap was being used)

=== 2020-12-22 ===
* 15:30 dcaro: cleaning up 6778 dangling snapshots for glance images in eqiad ([[phab:T270478|T270478]])
* 13:51 dcaro: merged patch to move wikidumpparse backups to cloudvirt1025 to free space on cloudvirt1026

=== 2020-12-19 ===
* 16:18 dcaro: gzipped a bunch of logs on cloudvirt1004 due to / being out of space
* 00:14 bstorm: truncated /var/log/debug.1 on cloudcontrol1003 which appears to be the exact same content as the user.log files anyway
* 00:10 bstorm: truncated /var/log/daemon.log.1 and the haproxy log
* 00:02 bstorm: truncated /var/log/messages.1 on cloudcontrol1003

=== 2020-12-18 ===
* 23:53 bstorm: truncated haproxy.log.1 on cloudcontrol1003
* 20:46 andrewbogott: setting  pg and pgp number to 4096 for eqiad1-compute as joachim thinks 8192 might be too much [[phab:T270305|T270305]]
* 17:09 dcaro: finished cleaning up the dangling snapshots from cloudvirt1026 ([[phab:T270478|T270478]])
* 17:08 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1026) ([[phab:T270478|T270478]])
* 17:06 dcaro: finished cleaning up the dangling snapshots from cloudvirt1025 ([[phab:T270478|T270478]])
* 17:05 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1025) ([[phab:T270478|T270478]])
* 17:00 dcaro: finished cleaning up the dangling snapshots from cloudvirt1021 ([[phab:T270478|T270478]])
* 16:58 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1021) ([[phab:T270478|T270478]])
* 16:56 dcaro: finished cleaning up the dangling snapshots from cloudvirt1022 ([[phab:T270478|T270478]])
* 16:55 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1022) ([[phab:T270478|T270478]])
* 16:54 dcaro: finished cleaning up the dangling snapshots from cloudvirt1023 ([[phab:T270478|T270478]])
* 16:51 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1023) ([[phab:T270478|T270478]])
* 16:47 dcaro: finished cleaning up the dangling snapshots from cloudvirt1024, freed ~12% of the capacity ([[phab:T270478|T270478]])
* 16:21 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1024) ([[phab:T270478|T270478]])
* 16:13 andrewbogott: setting  autoscale to 'off' for both ceph pools (eqiad1-compute and eqiad1-glance-images) because we like how things are set and the autoscaler does not
* 10:33 dcaro: purging rbd snapshots for image fc6fb78b-4515-4dcc-8254-{{Gerrit|591b9fe01762}} ([[phab:T270478|T270478]])

=== 2020-12-17 ===
* 22:17 andrewbogott: correction to above, set the pg and pgp to 1024 for eqiad1-glance-images
* 22:16 andrewbogott: setting  pgp number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) [[phab:T270305|T270305]] (same as pg)
* 22:14 andrewbogott: setting  pg number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) [[phab:T270305|T270305]]
* 22:10 andrewbogott: setting  autoscale to 'warn' for both ceph pools (eqiad1-compute and eqiad1-glance-images)

=== 2020-12-16 ===
* 09:31 dcaro: removing invalid backups from cloudvirt1024 (196 in total) ([[phab:T269419|T269419]])

=== 2020-12-14 ===
* 17:42 dcaro: The removal freed ~12GB (still 100% usage :S) ([[phab:T269419|T269419]])
* 17:36 dcaro: removing invalid backups that have a valid copy ([[phab:T269419|T269419]])
* 15:43 dcaro: Merging the tagging for vm backups ([[phab:T267195|T267195]])
* 09:45 arturo: icinga downtime cloudvirt1024 for 6 days ([[phab:T269419|T269419]])

=== 2020-12-13 ===
* 09:11 _dcaro: running backup purge script on cloudvirt1024 ([[phab:T269419|T269419]])

=== 2020-12-10 ===
* 23:36 bstorm: cleaned up the logs for haproxy on cloudcontrol1003 by deleting all the gzipped ones and truncating the .1 file
* 11:56 dcaro: Freed some space on cloudvirt1024 by running the purge script ([[phab:T269419|T269419]])
* 09:17 dcaro: removing leaked dns record discordwiki.eqiad.wmflabs (clinic duty)

=== 2020-12-08 ===
* 18:01 dcaro: Host cloudvirt1030 up and running ([[phab:T216195|T216195]])
* 15:59 dcaro: Re-imaging host cloudvirt1030 ([[phab:T216195|T216195]])
* 14:18 dcaro: Host online cloudvirt1029 ([[phab:T216195|T216195]])
* 14:13 dcaro: Host re-imaged, doing tests cloudvirt1029 ([[phab:T216195|T216195]])
* 12:14 dcaro: Re-imaging cloudvirt1029 ([[phab:T216195|T216195]])

=== 2020-12-07 ===
* 18:33 andrewbogott: putting cloudvirt1023 back into service [[phab:T269467|T269467]]
* 15:55 andrewbogott: reimaging cloudvirt1028 for [[phab:T216195|T216195]]
* 14:49 dcaro: Re-imaging cloudvirt1027 ([[phab:T216195|T216195]])

=== 2020-12-05 ===
* 00:35 andrewbogott: moving cloudvirt1023 back into maintenance because [[phab:T269467|T269467]] continues to puzzle

=== 2020-12-04 ===
* 22:33 andrewbogott: moving cloudvirt1023 back into the ceph aggregate; it doesn't need upgrades after all  [[phab:T269467|T269467]]
* 22:24 andrewbogott: moving cloudvirt1023 out of the ceph aggregate and into maintenance for [[phab:T269467|T269467]]
* 21:06 andrewbogott: putting cloudvirt1025 and 1026 back into service because I'm pretty sure they're fixed.  [[phab:T269313|T269313]]
* 12:12 arturo: manually running `wmcs-purge-backups` again on cloudvirt1024 ([[phab:T269419|T269419]])
* 11:25 arturo: icinga downtime cloudvirt1024 for 6 days, to avoid paging noises ([[phab:T269419|T269419]])
* 11:25 arturo: last log line referencing cloudvirt1024 is a mistake ([[phab:T269313|T269313]])
* 11:24 arturo: icinga downtime cloudvirt1024 for 6 days, to avoid paging noises ([[phab:T269313|T269313]])
* 10:28 arturo: manually running `wmcs-purge-backups` on cloudvirt1024 ([[phab:T269419|T269419]])
* 10:23 arturo: setting expiration to 2020-12-03 to the oldest backy snapshot of every VM in cloudvirt1024 ([[phab:T269419|T269419]])
* 09:54 arturo: icinga downtime cloudvirt1025 for 6 days ([[phab:T269313|T269313]])

=== 2020-12-03 ===
* 23:21 andrewbogott: removing all osds on cloudcephosd1004 for rebuild, [[phab:T268746|T268746]]
* 21:45 andrewbogott: removing all osds on cloudcephosd1005 for rebuild, [[phab:T268746|T268746]]
* 19:51 andrewbogott: removing all osds on cloudcephosd1006 for rebuild, [[phab:T268746|T268746]]
* 17:01 arturo: icinga downtime cloudvirt1025 for 48h to debug network issue [[phab:T269313|T269313]]
* 16:56 arturo: rebooting cloudvirt1025 to debug network issue [[phab:T269313|T269313]]
* 16:38 dcaro: Rimaging cloudvirt1026 ([[phab:T216195|T216195]])
* 13:24 andrewbogott: removing all osds on cloudcephosd1008 for rebuild, [[phab:T268746|T268746]]
* 02:55 andrewbogott: removing all osds on cloudcephosd1009 for rebuild, [[phab:T268746|T268746]]

=== 2020-12-02 ===
* 20:04 andrewbogott: removing all osds on cloudcephosd1010 for rebuild, [[phab:T268746|T268746]]
* 17:25 arturo: [15:51] failovering neutron virtual router in eqiad1 ([[phab:T268335|T268335]])
* 15:36 arturo: conntrackd is now up and running in cloudnet1003/1004 nodes ([[phab:T268335|T268335]])
* 15:33 arturo: [codfw1dev] conntrackd is now up and running in cloudnet200x-dev nodes ([[phab:T268335|T268335]])
* 15:08 andrewbogott: removing all osds on cloudcephosd1012 for rebuild, [[phab:T268746|T268746]]
* 12:41 arturo: disable puppet in all cloudnet servers to merge conntrackd change [[phab:T268335|T268335]]
* 11:12 dcaro: Reset the properties for the flavor g2.cores8.ram16.disk1120 to correct quotes ([[phab:T269172|T269172]])
* 09:57 arturo: moved cloudvirts 1030, 1029, 1028, 1027, 1026, 1025 away from the 'standard' host aggregate to 'maintenance' ([[phab:T269172|T269172]])

=== 2020-12-01 ===
* 20:06 andrewbogott: removing all osds on cloudcephosd1014 for rebuild, [[phab:T268746|T268746]]
* 12:04 arturo: restarting neutron l3 agents to pick up config change
* 11:48 arturo: merging change to dmz_dir, detail list of private address https://gerrit.wikimedia.org/r/c/operations/puppet/+/641977

=== 2020-11-30 ===
* 18:12 andrewbogott: removing all osds from cloudcephosd1015 in order to investigate [[phab:T268746|T268746]]

=== 2020-11-29 ===
* 17:18 andrewbogott: cleaning up some logfiles in tools-sgecron-01 — drive is full

=== 2020-11-26 ===
* 22:58 andrewbogott: deleting /var/log/haproxy logs older than 7 days in cloudcontrol100x.  We need log rotation here it seems.
* 15:53 dcaro: Created private flavor g2.cores8.ram16.disk1120 for wikidumpparse ([[phab:T268190|T268190]])

=== 2020-11-25 ===
* 19:35 bstorm: repairing ceph pg `instructing pg 6.91 on osd.117 to repair`
* 09:31 _dcaro: The OSD seems to be up and running actually, though there's that misleading log, will leave it see if the cluster comes fully healthy ([[phab:T268722|T268722]])
* 08:54 _dcaro: Unsetting noup/nodown to allow re-shuffling of the pgs that osd.44 had, will try to rebuild it ([[phab:T268722|T268722]])
* 08:45 _dcaro: Tried resetting the class for osd.44 to ssd, no luck, the cluster is in noout/norebalance to avoid data shuffling (opened [[phab:T268722|T268722]])
* 08:45 _dcaro: Tried resetting the class for osd.44 to ssd, no luck, the cluster is in noout/norebalance to avoid data shuffling (opened root@cloudcephosd1005:/var/lib/ceph/osd/ceph-44# ceph osd crush set-device-class ssd osd.44)
* 08:19 _dcaro: Restarting serivce osd.44 resulted on osd.44 being unable to start due to some config inconsistency (can not reset class to hdd)
* 08:16 _dcaro: After enabling auto pg scaling on ceph eqiad cluster, osd.44 (cloudcephosd1005) got stuck, trying to restart the osd service
* 08:16 _dcaro: After enabling auto pg scaling on ceph eqiad cluster, osd.44 (cloudcephosd1005) got stuck, trying to restart

=== 2020-11-22 ===
* 17:40 andrewbogott: apt-get upgrade on cloudservices1003/1004
* 17:32 andrewbogott: upgrading Designate on cloudservices1003/1004 to Stein

=== 2020-11-20 ===
* 12:44 arturo: [codfw1dev] install conntrackd in cloudnet2003-dev/cloudnet2002-dev to research l3 agent HA reliability
* 09:26 arturo: incinga downtime labstore1006 RAID checks for 10 days ([[phab:T268281|T268281]])

=== 2020-11-17 ===
* 19:21 andrewbogott: draining cloudvirt1012 to experiment with libvirt/cpu things

=== 2020-11-15 ===
* 11:21 arturo: icinga downtime cloudbackup2002 for 48h ([[phab:T267865|T267865]])

=== 2020-11-10 ===
* 16:38 arturo: icinga downtime toolschecker for 2h becasue toolsdb maintenance ([[phab:T266587|T266587]])
* 11:24 arturo: [codfw1dev] enable puppet in puppetmaster01.cloudinfra-codfw1dev (disabled for unspecified reasons)

=== 2020-11-09 ===
* 12:42 arturo: restarted neutron l3 agent in cloudnet1003 bc it still had the old default route ([[phab:T265288|T265288]])
* 12:41 arturo: `root@cloudcontrol1005:~# neutron subnet-delete dcbb0f98-5e9d-4a93-8dfc-4e3ec3c44dcc` ([[phab:T265288|T265288]])
* 12:41 arturo: `root@cloudcontrol1005:~# neutron router-gateway-set  --fixed-ip subnet_id=7c6bcc12-212f-44c2-9954-{{Gerrit|5c55002ee371}},ip_address=185.15.56.244 cloudinstances2b-gw wan-transport-eqiad` ([[phab:T265288|T265288]])
* 12:19 arturo: subnet 185.1.5.56.240/29 has id 7c6bcc12-212f-44c2-9954-{{Gerrit|5c55002ee371}} in neutron ([[phab:T265288|T265288]])
* 12:19 arturo: `root@cloudcontrol1005:~# neutron subnet-create --gateway 185.15.56.241 --name cloud-instances-transport1-b-eqiad1 --ip-version 4 --disable-dhcp wan-transport-eqiad 185.15.56.240/29` ([[phab:T265288|T265288]])
* 12:15 arturo: icinga-downtime toolschecker for 2h ([[phab:T265288|T265288]])

=== 2020-11-02 ===
* 13:36 arturo: (typo: dcaro)
* 13:35 arturo: added dcar as projectadmin &amp; user ([[phab:T266068|T266068]])

=== 2020-10-29 ===
* 16:57 bstorm: silenced deployment-prep project alerts for 60 days since the downtime expired
* 08:12 arturo: force-powercycling cloudcephosd1006

=== 2020-10-25 ===
* 16:20 andrewbogott: adding cloudvirt1038 to the 'ceph' aggregate and removing from the 'spare' aggregate.  We need this space while waiting on network upgrades for empty cloudvirts ([[phab:T216195|T216195]])

=== 2020-10-23 ===
* 11:30 arturo: [codfw1dev] openstack --os-project-id cloudinfra-codfw1dev recordset create --type PTR --record nat.cloudgw.codfw1dev.wikimediacloud.org. --description &quot;created by hand&quot; 0-29.57.15.185.in-addr.arpa. 1.0-29.57.15.185.in-addr.arpa. ([[phab:T261724|T261724]])
* 10:09 arturo: [codf1dev] doing DNS changes for the cloudgw PoC, including designate and https://gerrit.wikimedia.org/r/c/operations/dns/+/635965 ([[phab:T261724|T261724]])

=== 2020-10-22 ===
* 10:46 arturo: [codfw1dev] rebooting cloudinfra-internal-puppetmaster-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud to try fixing some DNS weirdness
* 09:43 arturo: enabling puppet in cloucontrol1003 (message said &quot;please re-enable after 2020-10-22 06:00UTC&quot;)

=== 2020-10-21 ===
* 14:36 andrewbogott: running apt-get update &amp;&amp; apt-get install -y facter on all cloud-vps instances
* 10:31 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])
* 08:56 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])

=== 2020-10-20 ===
* 15:47 arturo: changing DNS recursor ACLs (https://gerrit.wikimedia.org/r/c/operations/puppet/+/635314) this can be reverted any time if it causes problems ([[phab:T261724|T261724]])
* 14:49 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])

=== 2020-10-19 ===
* 01:41 andrewbogott: deleting all Precise base images
* 01:36 andrewbogott: deleting all unused Jessie base images

=== 2020-10-18 ===
* 23:26 andrewbogott: deleting all Trusty base images
* 21:50 andrewbogott: migrating all currently used ceph images to rbd

=== 2020-10-16 ===
* 09:29 arturo: [codfw1dev] still some DNS weirdness, investigating
* 09:25 arturo: [codfw1dev] hard-rebooting bastion-codfw1dev-02, seems in bad shape, doesn't even wake up in the virsh console
* 09:18 arturo: [codfw1dev] live-hacked cloudservices2002-dev /etc/powerdns/recursor.conf file to include cloud-codfw1dev-floating CIDR (185.15.57.0/29) while https://gerrit.wikimedia.org/r/c/operations/puppet/+/634050 is in review, so VMs with a floating IP can query the DNS recursor ([[phab:T261724|T261724]])
* 09:01 arturo: [codfw1dev] basic network connectivity seems stable after cleaning up everything related to address scopes ([[phab:T261724|T261724]])

=== 2020-10-15 ===
* 15:17 arturo: [codfw1dev] try cleaning up anything related to address scopes in the neutron database ([[phab:T261724|T261724]])
* 13:56 arturo: [codfw1dev] drop neutron l3 agent hacks in cloudnet2002/2003-dev ([[phab:T261724|T261724]])

=== 2020-10-13 ===
* 17:54 andrewbogott: rebuilding cloudvirt1021 for backy support
* 15:22 andrewbogott: draining cloudvirt1021 so I can rebuild it with backy support
* 14:19 andrewbogott: rebuilding cloudvirt1022 with backy support
* 14:03 andrewbogott: draining cloudvirt1022 so I can rebuild it with backy support
* 11:19 arturo: [codfw1dev] rebooting labtestvirt2003

=== 2020-10-09 ===
* 10:15 arturo: [codfwd1ev] root@cloudcontrol2001-dev:~# openstack router set --disable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw ([[phab:T261724|T261724]])
* 09:22 arturo: [codfwd1dev] rebooting cloudnet boxes for bridge and vlan changes ([[phab:T261724|T261724]])
* 09:12 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet delete 31214392-9ca5-4256-bff5-{{Gerrit|1e19a35661de}} (cloud-instances-transport1-b-codfw - 208.80.153.184/29) ([[phab:T261724|T261724]])
* 09:10 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack router set --external-gateway wan-transport-codfw --fixed-ip subnet=cloud-gw-transport-codfw,ip-address=185.15.57.10 cloudinstances2b-gw ([[phab:T261724|T261724]])
* 08:49 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.9 --no-dhcp --subnet-range 185.15.57.8/30 cloud-gw-transport-codfw ([[phab:T261724|T261724]])
* 08:47 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet delete a5ab5362-4ffb-4059-9ff7-{{Gerrit|391e22dcf3bc}} ([[phab:T261724|T261724]])

=== 2020-10-08 ===
* 16:17 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.8 --no-dhcp --subnet-range 185.15.57.8/31 cloud-gw-transport-codfw` (with a hack -- see task) ([[phab:T263622|T263622]])
* 16:03 arturo: [codfw1dev] briefly live-hacked python3-neutron source code in all 3 cloudcontrol2xxx-dev servers to workaround /31 network definition issue ([[phab:T263622|T263622]])
* 10:28 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) [[phab:T261724|T261724]]

=== 2020-10-06 ===
* 21:30 andrewbogott: moved cloudvirt1013 out of the 'ceph' aggregate and into the 'maintenance' aggregate for [[phab:T243414|T243414]]
* 21:29 andrewbogott: draining cloudvirt1013 for upgrade to 10G networking
* 14:45 arturo: icinga downtime every cloud* lab* host for 60 minutes for keystone maintenance

=== 2020-10-05 ===
* 17:40 bd808: `service uwsgi-labspuppetbackend restart` on cloud-puppetmaster-03 ([[phab:T264649|T264649]])

=== 2020-10-02 ===
* 11:05 arturo: [codfw1dev] restarting rabbitmq-server in all 3 control nodes, the l3 agent was misbehaving
* 09:16 arturo: [codfw1dev] trying the labtestvirt2003 (cloudgw) reimage again ([[phab:T261724|T261724]])

=== 2020-10-01 ===
* 16:06 arturo: rebooting cloudvirt1024 to validate changes to /etc/network/interfaces file
* 15:36 arturo: [codfw1dev] reimaging labtestvirt2003

=== 2020-09-30 ===
* 16:47 andrewbogott: rebooting cloudvir1032, 1033, 1034 for [[phab:T262979|T262979]]
* 13:28 arturo: enable puppet, reboot and pool back cloudvirt1031
* 13:27 arturo: extend icinga downtimes for another 120 mins
* 13:15 arturo: `aborrero@cloudcontrol1003:~$ sudo nova-manage placement sync_aggregates` after reading a hint in nova-api.log
* 13:02 arturo: rebooting cloudvirt1016 and moving it to the ceph host aggregate
* 12:55 arturo: rebooting cloudvirt1014 and moving it to the ceph host aggregate
* 12:51 arturo: rebooting cloudvirt1013 and moving it to the ceph host aggregate
* 12:39 arturo: root@cloudcontrol1005:~# openstack aggregate add host maintenance cloudvirt1031
* 12:36 arturo: rebooted cloudnet1003 (active) a couple of minutes ago
* 12:36 arturo: move cloudvirt1012 and cloudvirt1039 to the ceph aggregate
* 11:49 arturo: rebooting cloudvirt1039
* 11:46 arturo: rebooting cloudvirt1012
* 11:40 arturo: rebooting cloudnet1004 (standby) to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167 ([[phab:T262979|T262979]])
* 11:38 arturo: [codfw1dev] rebooting cloudnet2002-dev to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167
* 11:36 arturo: [codfw1dev] rebooting cloudnet2003-dev to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167
* 11:33 arturo: disabling puppet and downtiming every virt/net server in the fleet in preparation for merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167 ([[phab:T262979|T262979]])
* 09:32 arturo: rebooting cloudvirt1012 to investigate linuxbridge agent issues

=== 2020-09-29 ===
* 15:40 arturo: downgrade linux kernel from linux-image-4.19.0-11-amd64 to linux-image-4.19.0-10-amd64 on cloudvirt1012
* 14:47 arturo: rebooting cloudvirt1012, chasing config weirdness in the linuxbridge agent
* 14:05 andrewbogott: reimaging 1014 over and over in an attempt to get partman right
* 13:51 arturo: rebooting cloudvirt1012

=== 2020-09-28 ===
* 14:55 arturo: [jbond42] upgraded facter to v3 across the VM fleet
* 13:54 andrewbogott: moving cloudvirt1035 from aggregate 'spare' to 'ceph'.  We're going to need all the capacity we can get while converting older cloudvirts to ceph

=== 2020-09-24 ===
* 15:47 arturo: stopping/restarting rabbitmq-server in all cloudcontrol servers
* 15:45 arturo: restarting rabbitmq-server in cloudcontrol103
* 15:15 arturo: restarting floating_ip_ptr_records_updater.service in all 3 cloudcontrol servers to reset state after a DNS failure

=== 2020-09-18 ===
* 10:16 arturo: cloudvirt1039 libvirtd service issues were fixed with a reboot
* 09:56 arturo: rebooting cloudvirt1039 (spare) to try to fix some weird libvirtd failure
* 09:50 arturo: enabling puppet in cloudvirts and effectively merging patches from [[phab:T262979|T262979]]
* 08:59 arturo: disable puppet in all buster cloudvirts (cloudvirt[1024,1031-1039].eqiad.wmnet) to merge a patch for [[phab:T263205|T263205]] and [[phab:T262979|T262979]]
* 08:50 arturo: installing iptables from buster-bpo in cloudvirt1036 ([[phab:T263205|T263205]] and [[phab:T262979|T262979]])

=== 2020-09-15 ===
* 20:32 andrewbogott: rebooting cloudvirt1038 to see if it resolves [[phab:T262979|T262979]]
* 13:58 andrewbogott: draining cloudvirt1002 with wmcs-ceph-migrate

=== 2020-09-14 ===
* 14:21 andrewbogott: draining cloudvirt1001, migrating all VMs with wmcs-ceph-migrate
* 10:41 arturo: [codfw1dev] trying to get the bonding working for labtestvirt2003 ([[phab:T261724|T261724]])
* 09:47 arturo: installed qemu security update in eqiad1 cloudvirts ([[phab:T262386|T262386]])
* 09:43 arturo: [codfw1dev] installed qemu security update in codfw1dev cloudvirts ([[phab:T262386|T262386]])

=== 2020-09-09 ===
* 18:13 andrewbogott: restarting ceph-mon@cloudcephmon1003 in hopes that the slow ops reported are phantoms
* 18:01 andrewbogott: restarting ceph-mgr@cloudcephmon1003 in hopes that the slow ops reported are phantoms (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/EOWNO3MDYRUZKAK6RMQBQ5WBPQNLHOPV/)
* 17:40 andrewbogott: giving ceph pg autoscale another chance: ceph osd pool set eqiad1-compute pg_autoscale_mode on
* 00:05 bd808: Running wmcs-novastats-dnsleaks ([[phab:T262359|T262359]])

=== 2020-09-08 ===
* 21:48 bd808: Renamed FQDN prefixes to wikimedia.cloud scheme in cloudinfra-db01's labspuppet db ([[phab:T260614|T260614]])
* 14:29 andrewbogott: restarting nova-compute on all cloudvirts (everyone is upset from the reset switch failure)
* 14:18 arturo: restarting nova-fullstack service in cloudcontrol1003
* 14:17 andrewbogott: stopping apache2 on labweb1001 to make sure the Horizon outage is total

=== 2020-09-03 ===
* 09:31 arturo: icinga downtime cloud* servers for 30 mins ([[phab:T261866|T261866]])

=== 2020-09-02 ===
* 08:46 arturo: [codfw1dev] reimaging spare server labtestvirt2003 as debian buster ([[phab:T261724|T261724]])

=== 2020-09-01 ===
* 18:18 andrewbogott: adding drives on cloudcephosd100[3-5] to ceph osd pool
* 13:40 andrewbogott: adding drives on cloudcephosd101[0-2] to ceph osd pool
* 13:35 andrewbogott: adding drives on cloudcephosd100[1-3] to ceph osd pool
* 11:27 arturo: [codfw1dev] rebooting again cloudnet2002-dev after some network tests, to reset initial state ([[phab:T261724|T261724]])
* 11:09 arturo: [codfw1dev] rebooting cloudnet2002-dev after some network tests, to reset initial state ([[phab:T261724|T261724]])
* 10:49 arturo: disable puppet in cloudnet servers to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/623569/

=== 2020-08-31 ===
* 23:26 bd808: Removed stale lockfile at cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs:/var/lib/puppet/volatile/GeoIP/.geoipupdate.lock
* 11:20 arturo: [codfw1dev] livehacking https://gerrit.wikimedia.org/r/c/operations/puppet/+/615161 in the puppetmasters for tests before merging

=== 2020-08-28 ===
* 20:12 bd808: Running `wmcs-novastats-dnsleaks --delete` from cloudcontrol1003

=== 2020-08-26 ===
* 17:12 bstorm: Running 'ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf &quot;%k KB %p\n&quot; &gt; tools_large_files_20200826.txt' on labstore1004 [[phab:T261336|T261336]]

=== 2020-08-21 ===
* 21:34 andrewbogott: restarting nova-compute on cloudvirt1033; it seems stuck

=== 2020-08-19 ===
* 14:21 andrewbogott: rebooting cloudweb2001-dev, labweb1001, labweb1002 to address mediawiki-induced memleak

=== 2020-08-06 ===
* 21:02 andrewbogott: removing cloudvirt1004/1006 from nova's list of hypervisors; rebuilding them to  use as backup test hosts
* 20:06 bstorm: manually stopped the RAID check on cloudcontrol1003 [[phab:T259760|T259760]]

=== 2020-08-04 ===
* 18:54 bstorm: restarting mariadb on cloudcontrol1004 to setup parallel replication

=== 2020-08-03 ===
* 17:02 bstorm: increased db connection limit to 800 across galera cluster because we were clearly hovering at limit

=== 2020-07-31 ===
* 19:28 bd808: wmcs-novastats-dnsleaks --delete (lots of leaked fullstack-monitoring records to clean up)

=== 2020-07-27 ===
* 22:17 andrewbogott: ceph osd pool set compute pg_num 2048
* 22:14 andrewbogott: ceph osd pool set compute pg_autoscale_mode off

=== 2020-07-24 ===
* 19:15 andrewbogott: ceph mgr module enable pg_autoscaler
* 19:15 andrewbogott: ceph osd pool set compute pg_autoscale_mode on

=== 2020-07-22 ===
* 08:55 jbond42: [codfw1dev] upgrading hiera to version5
* 08:48 arturo: [codfw1dev] add jbond as user in the bastion-codfw1dev and cloudinfra-codfw1dev projects
* 08:45 arturo: [codfw1dev] enabled account creation in labtestwiki briefly for jbond42 to create an account

=== 2020-07-16 ===
* 10:48 arturo: merging change to neutron dmz_cidr https://gerrit.wikimedia.org/r/c/operations/puppet/+/613123 ([[phab:T257534|T257534]])

=== 2020-07-15 ===
* 23:15 bd808: Removed Merlijn van Deen from toollabs-trusted Gerrit group ([[phab:T255697|T255697]])
* 11:48 arturo: [codfw1dev] created DNS records (A and PTR) for bastion.bastioninfra-codfw1dev.codfw1dev.wmcloud.org &lt;-&gt; 185.15.57.2
* 11:41 arturo: [codfw1dev] add myself as projectadmin to the `bastioninfra-codfw1dev` project
* 11:39 arturo: [codfw1dev] created DNS zone `bastioninfra-codfw1dev.codfw1dev.wmcloud.org.` in the cloudinfra-codfw1dev project and then transfer ownership to the bastioninfra-codfw1dev project

=== 2020-07-14 ===
* 15:19 arturo: briefly set root@cloudnet1003:~ # sysctl net.ipv4.conf.all.accept_local=1 (in neutron qrouter netns) ([[phab:T257534|T257534]])
* 10:43 arturo: icinga downtime cloudnet* hosts for 30 mins to introduce new check https://gerrit.wikimedia.org/r/c/operations/puppet/+/612390 ([[phab:T257552|T257552]])
* 04:01 andrewbogott: added a wildcard *.wmflabs.org domain pointing at the domain proxy in project-proxy
* 04:00 andrewbogott: shortened the ttl on .wmflabs.org. to 300

=== 2020-07-13 ===
* 16:17 arturo: icinga downtime cloudcontrol[1003-1005].wikimedia.org for 1h for galera database movements

=== 2020-07-12 ===
* 17:39 andrewbogott: switched eqiad1 keystone from m5 to cloudcontrol galera

=== 2020-07-10 ===
* 20:26 andrewbogott: disabling nova api to move database to galera

=== 2020-07-09 ===
* 11:23 arturo: [codfw1dev] rebooting cloudnet2003-dev again for testing sysct/puppet behavior ([[phab:T257552|T257552]])
* 11:11 arturo: [codfw1dev] rebooting cloudnet2003-dev for testing sysct/puppet behavior ([[phab:T257552|T257552]])
* 09:16 arturo: manually increasing sysctl value of net.nf_conntrack_max in cloudnet servers ([[phab:T257552|T257552]])

=== 2020-07-06 ===
* 15:16 arturo: installing 'aptitude' in all cloudvirts

=== 2020-07-03 ===
* 12:51 arturo: [codfw1dev] galera cluster should be up and running, openstack happy ([[phab:T256283|T256283]])
* 11:44 arturo: [codfw1dev] restoring glance database backup from bacula into cloudcontrol2001-dev ([[phab:T256283|T256283]])
* 11:39 arturo: [codfw1dev] stopped mysql database in the galera cluster [[phab:T256283|T256283]]
* 11:36 arturo: [codfw1dev] dropped glance database in the galera cluster [[phab:T256283|T256283]]

=== 2020-07-02 ===
* 15:41 arturo: `sudo wmcs-openstack --os-compute-api-version 2.55 flavor create --private --vcpus 8 --disk 300 --ram 16384 --property aggregate_instance_extra_specs:ceph=true --description &quot;for packaging envoy&quot; bigdisk-ceph` ([[phab:T256983|T256983]])

=== 2020-06-29 ===
* 14:24 arturo: starting rabbitmq-server in all 3 cloudcontrol servers
* 14:23 arturo: stopping rabbitmq-server in all 3 cloudcontrol servers

=== 2020-06-18 ===
* 20:38 andrewbogott: rebooting cloudservices2003-dev due to a mysterious 'host down' alert on a secondary ip

=== 2020-06-16 ===
* 15:38 arturo: created by hand neutron port 9c0a9a13-e409-49de-9ba3-{{Gerrit|bc8ec4801dbf}} `paws-haproxy-vip` ([[phab:T295217|T295217]])

=== 2020-06-12 ===
* 13:23 arturo: DNS zone `paws.wmcloud.org` transferred to the PAWS project ([[phab:T195217|T195217]])
* 13:20 arturo: created DNS zone `paws.wmcloud.org` ([[phab:T195217|T195217]])

=== 2020-06-11 ===
* 19:19 bstorm_: proceeding with failback to labstore1004 now that DRBD devices are consistent [[phab:T224582|T224582]]
* 17:22 bstorm_: delaying failback labstore1004 for drive syncs [[phab:T224582|T224582]]
* 17:17 bstorm_: failing NFS back to labstore1004 to complete the upgrade process [[phab:T224582|T224582]]
* 16:15 bstorm_: failing over NFS for labstore1004 to labstore1005 [[phab:T224582|T224582]]

=== 2020-06-10 ===
* 16:09 andrewbogott: deleting all old cloud-ns0.wikimedia.org and cloud-ns1.wikimedia.org ns records in designate database [[phab:T254496|T254496]]

=== 2020-06-09 ===
* 15:25 arturo: icinga downtime everything cloud* lab* for 2h more ([[phab:T253780|T253780]])
* 14:09 andrewbogott: stopping puppet, all designate services and all pdns services on cloudservices1004 for [[phab:T253780|T253780]]
* 14:01 arturo: icinga downtime everything cloud* lab* for 2h ([[phab:T253780|T253780]])

=== 2020-06-05 ===
* 15:08 andrewbogott: trying to re-enable puppet without losing cumin contact, as per https://phabricator.wikimedia.org/T254589

=== 2020-06-04 ===
* 14:24 andrewbogott: disabling puppet on all instances for /labs/private recovery
* 14:23 arturo: disabling puppet on all instances for /labs/private recovery

=== 2020-05-28 ===
* 23:02 bd808: `/usr/local/sbin/maintain-dbusers --debug harvest-replicas` ([[phab:T253930|T253930]])
* 13:36 andrewbogott: rebuilding cloudservices2002-dev with Buster
* 00:33 andrewbogott: shutting down cloudservices2002-dev to see if we can live without it.  This is in anticipation or rebuilding it entirely for [[phab:T253780|T253780]]

=== 2020-05-27 ===
* 23:29 andrewbogott: disabling the backup job on cloudbackup2001 (just like last week) so the backup doesn't start while Brooke is rebuilding labstore1004 tomorrow.
* 06:03 bd808: `systemctl start mariadb` on clouddb1001 following reboot (take 2)
* 05:58 bd808: `systemctl start mariadb` on clouddb1001 following reboot
* 05:53 bd808: Hard reboot of clouddb1001 via Horizon. Console unresponsive.

=== 2020-05-25 ===
* 16:35 arturo: [codfw1dev] created zone `0-29.57.15.185.in-addr.arpa.` ([[phab:T247972|T247972]])

=== 2020-05-21 ===
* 19:23 andrewbogott: disabling puppet on cloudbackup2001 to prevent the backup job from starting during maintenance
* 19:16 andrewbogott: systemctl disable block_sync-tools-project.service on cloudbackup2001.codfw.wmnet to avoid stepping on current upgrade
* 15:48 andrewbogott: re-imaging cloudnet1003 with Buster

=== 2020-05-19 ===
* 22:59 bd808: `apt-get install mariadb-client` on cloudcontrol1003
* 21:12 bd808: Migrating wcdo.wcdo.eqiad.wmflabs to cloudvirt1023 ([[phab:T251065|T251065]])

=== 2020-05-18 ===
* 21:37 andrewbogott: rebuilding cloudnet2003-dev with Buster

=== 2020-05-15 ===
* 22:10 bd808: Added reedy as projectadmin in cloudinfra project ([[phab:T249774|T249774]])
* 22:05 bd808: Added reedy as projectadmin in admin project ([[phab:T249774|T249774]])
* 18:44 bstorm_: rebooting cloudvirt-wdqs1003 [[phab:T252831|T252831]]
* 15:47 bd808: Manually running wmcs-novastats-dnsleaks from cloudcontrol1003 ([[phab:T252889|T252889]])

=== 2020-05-14 ===
* 23:28 bstorm_: downtimed cloudvirt1004/6 and cloudvirt-wdqs1003 until tomorrow around this time [[phab:T252831|T252831]]
* 22:21 bstorm_: upgrading qemu-system-x86 on cloudvirt1006 to backports version [[phab:T252831|T252831]]
* 22:15 bstorm_: changing /etc/libvirt/qemu.conf and restarting libvirtd on cloudvirt1006 [[phab:T252831|T252831]]
* 21:12 andrewbogott: rebuilding cloudvirt1003-wdqs as part of [[phab:T252831|T252831]]
* 15:47 andrewbogott: moving cloudvirt1004 and cloudvirt1006 to the 'ceph' aggregate for [[phab:T252784|T252784]]
* 15:02 andrewbogott: moving all of cloudvirt100[1-9] into the 'toobusy' host aggregate.  These are slower, have spinning disks, and are due for replacement.

=== 2020-05-12 ===
* 20:33 andrewbogott: moving cloudvirt1023 to the 'standard' pool and out of the 'spare' pool
* 19:10 jeh: disable neutron-openvswitch-agent service on cloudvirt2001-dev.codfw [[phab:T248881|T248881]]
* 19:09 jeh: Shutdown the unused eno2 network interface on cloudvirt2001-dev.codfw to clear up monitoring errors [[phab:T248425|T248425]]
* 18:20 andrewbogott: moving cloudvirt1024 out of the 'maintenance' aggregate and into 'spare'
* 16:45 andrewbogott: restarting neutron-l3-agent on cloudnet1004 so it knows about all three cloudcontrols.  Leaving cloudnet1003 since restarting it there will cause network interruptions
* 14:06 arturo: icinga downtime everything for 2h for Debian Buster migration in some cloud components

=== 2020-05-09 ===
* 16:53 andrewbogott: rebuilding cloudcontrol2001-dev and 2003-dev with buster for [[phab:T252121|T252121]]

=== 2020-05-08 ===
* 19:02 bstorm_: moving tools-k8s-haproxy-2 from cloudvirt1021 to cloudvirt1017 to improve spread

=== 2020-05-05 ===
* 13:58 andrewbogott: rebuilding cloudcontrol2004-dev to test new puppet changes

=== 2020-05-04 ===
* 09:04 arturo: [codfw1dev] manually modify iptables ruleset to only allow SSH from WMF bastions on cloudservices2003-dev and cloudcontrol2004-dev ([[phab:T251604|T251604]])

=== 2020-04-21 ===
* 22:12 andrewbogott: moving cloudvirt1004 out of the 'standard' aggregate and into the 'maintenance' aggregate
* 16:01 jeh: restart cloudceph mon and osd services for openssl upgrades

=== 2020-04-15 ===
* 18:44 jeh: create indexes and views for grwikimedia [[phab:T245912|T245912]]

=== 2020-04-13 ===
* 15:07 jeh: restart memcached on labwebs to increase cache size [[phab:T145703|T145703]]

=== 2020-04-09 ===
* 19:57 andrewbogott: upgrading eqiad1 designate to rocky
* 16:52 andrewbogott: cleaned up a bunch of leaked .eqiad.wmflabs dns records

=== 2020-04-08 ===
* 19:20 andrewbogott: rotated password and api token for pdns servers on cloudservices1003 and cloudservices1004
* 14:54 arturo: `root@cloudcontrol1003:~# cp /etc/inputrc .inputrc` to solve some bash shortcut weirdness

=== 2020-04-07 ===
* 20:57 andrewbogott: service sssd stop; rm -rf /var/lib/sss/db*; service sssd start on tools-sgebastion-08

=== 2020-04-06 ===
* 22:39 andrewbogott: deleting bogus groups cn=b'project-bastion',ou=groups,dc=wikimedia,dc=org and cn=b'project-tools',ou=groups,dc=wikimedia,dc=org from ldap
* 17:42 arturo: [codfw1dev] transferred DNS zone 57.15.185.in-addr.arpa. to the cloudinfra-codfw1dev project ([[phab:T247972|T247972]])
* 17:39 arturo: [codfw1dev] `openstack zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description &quot;floating IPs subnet&quot; 57.15.185.in-addr.arpa.` ([[phab:T247972|T247972]])
* 16:23 arturo: restarting apache2 in cloudcontrol1003/1004 to pick up latest wmfkeystonehooks changes [[phab:T249494|T249494]]

=== 2020-04-02 ===
* 20:59 jeh: codfw1dev clear VM error states and start bastions, puppet master and database

=== 2020-04-01 ===
* 16:27 arturo: [codfw1dev] enable puppet across the fleet clean vxlan changes ([[phab:T248881|T248881]])

=== 2020-03-31 ===
* 12:35 arturo: [codfw1dev] restarting VMs: designaterockytest14, bastion-codfw1dev-0[1,2] ([[phab:T248881|T248881]])
* 12:34 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudvirt2001-dev ([[phab:T248881|T248881]])
* 12:25 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudnet200[2,3]-dev ([[phab:T248881|T248881]])
* 11:45 arturo: [codfw1dev] rebooting cloudvirt2003-dev to pick up latest kernel update. Otherwise modprobe is confused trying to load modules and openvswitch won't start ([[phab:T248881|T248881]])
* 10:40 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudvirt2003-dev ([[phab:T248881|T248881]])
* 10:09 arturo: [codfw1dev] reboot cloudnet2003-dev into linux 4.9 (was using 4.14 from a testing operation in 2020-03-10)

=== 2020-03-30 ===
* 23:42 bstorm_: deleted &quot;Kubernetes Cluster&quot; and &quot;Kubernetes Performance&quot; dashboards [[phab:T246689|T246689]]
* 16:44 arturo: [codfw1dev] installing package neutron-openvswitch-agent in cloudvirt2002-dev ([[phab:T248881|T248881]])
* 16:42 andrewbogott: restarting l3 agents on cloudnets in codfw1dev after applying https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/584188/

=== 2020-03-27 ===
* 21:28 bd808: Created huggle.wmcloud.org Designate zone and allocated it to the huggle project
* 19:51 jeh: start haproxy on cloudcontrol2003-dev.wikimedia.org

=== 2020-03-26 ===
* 15:01 arturo: icinga downtime cloudvirt* cloudcontrol* cloudnet* lab* cloudstore*
* 15:01 andrewbogott: beginning openstack upgrade window for [[phab:T242766|T242766]]
* 12:32 arturo: [codfw1dev] downgraded systemd, libsystemd0, udev and friends to the non-backports versions ([[phab:T247013|T247013]])

=== 2020-03-25 ===
* 19:29 andrewbogott: dumping a bunch of VMs on cloudvirt1015 to see if it still crashes
* 17:56 jeh: add labweb1002 back into the pool - completed horizon testing [[phab:T240852|T240852]]
* 17:09 jeh: depool labweb1002 for horizon testing [[phab:T240852|T240852]]

=== 2020-03-24 ===
* 19:41 jeh: switch cloudvirt1016 from maintenance to standard host aggregate [[phab:T243327|T243327]]
* 15:31 andrewbogott: restarting nova-conductor and nova-api on cloudcontrol1003 and cloudcontrol1004

=== 2020-03-23 ===
* 21:41 jeh: restart neutron-l3-agent on cloudnet100[3,4] to pickup policy.yaml changes
* 13:28 jeh: disable puppet on labweb100[1,2] to enable horizon event traces [[phab:T240852|T240852]]
* 10:26 arturo: restarting apache in both labweb1001/labweb1002 upon reports of returning 500s

=== 2020-03-21 ===
* 14:23 andrewbogott: restarting apache2 on labweb1001 and 1002

=== 2020-03-18 ===
* 19:17 andrewbogott: deleted a bunch of records from the pdns database on cloudservices1003/1004 which had a record name but the content (where an IP address should be) was NULL, e.g. m.wikidata.beta.wmflabs.org.
* 10:55 arturo: [codfw1dev] deleting BGP agent, undoing changes we did for [[phab:T245606|T245606]]

=== 2020-03-14 ===
* 17:40 jeh: restart maintain-dbusers on labstore1004 [[phab:T247654|T247654]]

=== 2020-03-13 ===
* 12:39 arturo: [codfw1dev] reintroduce address scopes for another round of testing [[phab:T244851|T244851]]
* 12:17 arturo: [codfw1dev] enabling puppet in cloudnet200x-dev servers after merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/579259 ([[phab:T247505|T247505]])

=== 2020-03-12 ===
* 22:29 bstorm_: running puppet across all dumps mounts to make sure active links are shifted to labstore1006

=== 2020-03-11 ===
* 18:38 jeh: set icingia downtime until 2020-03-23 on CODFW cloud[control,net,virt] hosts during openstack upgrades
* 12:50 arturo: [codfw1dev] several tests creating/deleting address scopes ([[phab:T244727|T244727]] [[phab:T247135|T247135]] [[phab:T246887|T246887]] [[phab:T245606|T245606]])
* 12:46 arturo: [codfw1dev] disable routing_source_ip in l3 agents for testing proposal detailed at https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Network_refresh#Eliminate_routing_source_ip_address ([[phab:T244727|T244727]])

=== 2020-03-10 ===
* 17:02 arturo: [codfw1dev] deleting address scopes, bad interaction with our custom NAT setup [[phab:T247135|T247135]]
* 13:55 arturo: [codfw1dev] rebooting cloudnet2003-dev into linux kernel 4.14 for testing stuff related to [[phab:T247135|T247135]]

=== 2020-03-09 ===
* 18:09 arturo: enabling puppet in cloudvirt1006, all services have been restored
* 17:59 arturo: deleted the neutron bridge on cloudvirt1006, for testing stuff related to the queens upgrade
* 17:58 arturo: stopped neutron-linuxbridge-agent and nova-compute in cloudvirt1006 for testing stuff related to the queens upgrade

=== 2020-03-06 ===
* 14:54 andrewbogott: draining all instances off of cloudvirt1006 for [[phab:T246908|T246908]]

=== 2020-03-05 ===
* 14:24 arturo: [codfw1dev] we just enabled BGP session between cloudnet2xxx-dev and cr1-codfw ([[phab:T245606|T245606]])
* 13:07 arturo: [codfw1dev] move the extra IP address for BGP in cloudnet200x-dev servers from eno2.2120 to the br-external bridge device ([[phab:T245606|T245606]])
* 13:06 arturo: [codfw1dev] upgrade neutron-dynamic-routing packages in cloudnet200X-dev and cloudcontrol200X-dev servers to 11.0.0-2~bpo9+1 ([[phab:T245606|T245606]])

=== 2020-03-04 ===
* 22:22 andrewbogott: upgrading designate on cloudservices1003/1004 to Queens
* 22:09 andrewbogott: moving cloudvirt1006 into the maintenance aggregate for [[phab:T246908|T246908]]
* 21:37 bd808: Running wmcs-wikireplica-dns to add service names for ngwikimedia.*.db.svc.eqiad.wmflabs ([[phab:T240772|T240772]])
* 21:14 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1009 ([[phab:T246056|T246056]])
* 21:11 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1010 ([[phab:T246056|T246056]])
* 21:08 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1011 ([[phab:T246056|T246056]])
* 21:05 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1002 ([[phab:T246056|T246056]])

=== 2020-03-02 ===
* 16:54 arturo: [codfw1dev] deleted python3-os-ken debian package in cloudnet2003-dev which was installed by hand and had depedency issues

=== 2020-02-29 ===
* 16:32 bstorm_: downtimed the smart alert on cloudvirt1009 until Monday since apparently predictive failures flap [[phab:T244986|T244986]]

=== 2020-02-26 ===
* 22:03 jeh: powering down cloudvirt1014 for hardware maintenance

=== 2020-02-25 ===
* 16:08 andrewbogott: changing neutron's rabbitmq password because oslo is having trouble parsing some of the characters in the password
* 15:26 andrewbogott: updated the cell_mapping record in the nova_api database to add the second rabbitmq server  to the transport_url field
* 15:26 andrewbogott: updated the cell_mapping record in the nova_api database to set the db uri to 'mysql+pymysql' -- this in response to a deprecation notice

=== 2020-02-24 ===
* 12:16 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker cr2-codfw` ([[phab:T245606|T245606]])
* 12:16 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker cr1-codfw` ([[phab:T245606|T245606]])
* 12:09 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.187 --remote-as 65002 cr2-codfw` ([[phab:T245606|T245606]])
* 12:09 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.186 --remote-as 65002 cr1-codfw` ([[phab:T245606|T245606]])
* 12:06 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-delete 17b8c2a3-f0ce-4d50-a265-18ccac703c61` ([[phab:T245606|T245606]])
* 10:59 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker bgppeer` ([[phab:T245606|T245606]])
* 10:56 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.185 --remote-as 65002 bgppeer` ([[phab:T245606|T245606]])

=== 2020-02-21 ===
* 12:48 arturo: [codfw1dev] running `root@cloudcontrol2001-dev:~# neutron bgp-speaker-network-add bgpspeaker wan-transport-codfw` ([[phab:T245606|T245606]])
* 12:46 arturo: [codfw1dev] created bgpspeaker for AS64711 ([[phab:T245606|T245606]])
* 12:42 arturo: [codfw1dev] run `sudo neutron-db-manage upgrade head` to upgrade the db schema for neutron bgp tables
* 11:51 arturo: [codfw1dev] create a neutron subnet pool per each subnet objects we have and manually update DB to inter-associate them ([[phab:T245606|T245606]])
* 11:49 arturo: [codfw1dev] rename neutron address scope `no-nat` to `bgp` ([[phab:T245606|T245606]])
* 11:37 arturo: [codfw1dev] cleanup unused neutron subnet pools from previous address scope tests ([[phab:T244851|T244851]])

=== 2020-02-20 ===
* 19:22 andrewbogott: updating designate pool config for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/572213/
* 15:33 andrewbogott: migrating all VMs on cloudvirt1014 to cloudvirt1022
* 13:35 arturo: [codfw1dev] disable puppet in cloudcontrol servers to hack neutron.conf for tests related to [[phab:T245606|T245606]]
* 13:33 arturo: [codfw1dev] disable puppet in cloudnet servers to hack neutron.conf for tests related to [[phab:T245606|T245606]]

=== 2020-02-18 ===
* 22:19 andrewbogott: transferred the tools.wmcloud.org. to the tools project
* 22:16 andrewbogott: moved wmcloud.org dns domain to the cloud-infra project
* 21:02 andrewbogott: adding .eqiad1.wikimedia.cloud records to all existing eqiad1 VMs, updating all eqiad1 internal pointer records to reference the new eqiad1.wikimedia.cloud fqdns.
* 09:44 arturo: deleted DNS zone wmcloud.org and try re-creating it

=== 2020-02-14 ===
* 10:35 arturo: running `root@cloudcontrol2001-dev:~# designate server-create --name ns1.openstack.codfw1dev.wikimediacloud.org.` ([[phab:T243766|T243766]])
* 10:32 arturo: running `root@cloudcontrol1004:~# designate server-create --name ns1.openstack.eqiad1.wikimediacloud.org.` ([[phab:T243766|T243766]])
* 10:32 arturo: running `root@cloudcontrol1004:~# designate server-create --name ns0.openstack.eqiad1.wikimediacloud.org.` ([[phab:T243766|T243766]])

=== 2020-02-12 ===
* 13:38 arturo: [codfw1dev] add reference to subnetpool to the instance subnet `MariaDB [neutron]&gt; update subnets set subnetpool_id='d129650d-d4be-4fe1-b13e-6edb5565cb4a' where id = '7adfcebe-b3d0-4315-92fe-e8365cc80668';` ([[phab:T244851|T244851]])

=== 2020-02-11 ===
* 13:46 arturo: [codfw1dev] creating some neutron objects to investigate [[phab:T244851|T244851]] (subnets, subnet pools, address scopes, ...)
* 12:40 arturo: [codfw1dev] delete unknown address scope 'wmcs-v4-scope': `root@cloudcontrol2001-dev:~# openstack address scope delete 078cfd71-117b-4aac-9197-6ebbbb7dd3de` ([[phab:T244851|T244851]])
* 12:40 arturo: [codfw1dev] delete unknown subnet pool 'cloudinstancesb-v4-pool0': `root@cloudcontrol2001-dev:~# openstack subnet pool delete d23a9b88-5c3d-4a53-ab88-053233a75365` ([[phab:T244851|T244851]])

=== 2020-02-07 ===
* 18:11 jeh: shutdown cloudvirt1016 for hardware maintenance [[phab:T241882|T241882]]

=== 2020-02-06 ===
* 14:44 jeh: update apt packages on cloudvirt1015 [[phab:T220853|T220853]]
* 14:28 jeh: run hardware tests on cloudvirt1015 [[phab:T220853|T220853]]

=== 2020-01-28 ===
* 17:24 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# designate server-create --name ns0.openstack.codfw1dev.wikimediacloud.org. ([[phab:T243766|T243766]])
* 10:18 arturo: [codfw1dev] created DNS record `bastion-codfw1dev-01.codfw1dev.wmcloud.org A 185.15.57.2` ([[phab:T242976|T242976]], [[phab:T229441|T229441]])
* 10:13 arturo: [codfw1dev] the zone `codfw1dev.wmcloud.org` belongs now to the `cloudinfra-codfw1dev` project ([[phab:T242976|T242976]])
* 10:11 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description &quot;main DNS domain for public addresses&quot; --email &quot;root@wmflabs.org&quot; --type PRIMARY --ttl 3600 codfw1dev.wmcloud.org.` ([[phab:T242976|T242976]] and [[phab:T243766|T243766]])
* 09:53 arturo: restart apache2 in labweb1001/1002 because horizon errors
* 09:47 arturo: created DNS zone wmcloud.org in eqiad1, transfer it to the cloudinfra project ([[phab:T242976|T242976]]) right now only use is to delegate codfw1dev.wmcloud.org subdomain to designate in the other deployment

=== 2020-01-27 ===
* 12:45 arturo: [codfw1dev] manually move the new domain to the `cloudinfra-codfw1dev` project clouddb2001-dev: `[designate]&gt; update zones set tenant_id='cloudinfra-codfw1dev' where id = '4c75410017904858a5839de93c9e8b3d';` [[phab:T243556|T243556]]
* 12:44 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description &quot;main DNS domain for VMs&quot; --email &quot;root@wmflabs.org&quot; --type PRIMARY --ttl 3600 codfw1dev.wikimedia.cloud.` [[phab:T243556|T243556]]

=== 2020-01-24 ===
* 15:10 jeh: remove icinga downtime for cloudvirt1013 [[phab:T241313|T241313]]
* 12:52 arturo: repooling cloudvirt1013 after HW got fixed ([[phab:T241313|T241313]])

=== 2020-01-21 ===
* 17:43 bstorm_: remounting /mnt/nfs/dumps-labstore1007.wikimedia.org/ on all dumps-mounting projects
* 10:24 arturo: running `sudo systemctl restart apache2.service` in both labweb servers to try mitigating [[phab:T240852|T240852]]

=== 2020-01-15 ===
* 16:59 bd808: Changed the config for cloud-announce mailing list so that lsit admins do not get bounce unsubscribe notices

=== 2020-01-14 ===
* 14:03 arturo: icinga downtime all cloudvirts for another 2h for fixing some icinga checks
* 12:04 arturo: icinga downtime toolchecker for 2 hours for openstack upgrades [[phab:T241347|T241347]]
* 12:02 arturo: icinga downtime cloud* labs* hosts for 2 hours for openstack upgrades [[phab:T241347|T241347]]
* 04:26 andrewbogott: upgrading designate on cloudservices1003/1004

=== 2020-01-13 ===
* 13:34 arturo: [¢odfw1dev] prevent neutron from allocating floating IPs from the wrong subnet by doing `neutron subnet-update --allocation-pool start=208.80.153.190,end=208.80.153.190 cloud-instances-transport1-b-codfw` ([[phab:T242594|T242594]])

=== 2020-01-10 ===
* 13:27 arturo: cloudvirt1009: virsh undefine i-000069b6. This is tools-elastic-01 which is running on cloudvirt1008 (so, leaked on cloudvirt1009)

=== 2020-01-09 ===
* 11:12 arturo: running `MariaDB [nova_eqiad1]&gt; update quota_usages set in_use='0' where project_id='etytree';` ([[phab:T242332|T242332]])
* 11:11 arturo: running `MariaDB [nova_eqiad1]&gt; select * from quota_usages where project_id = 'etytree';` ([[phab:T242332|T242332]])
* 10:32 arturo: ran `root@cloudcontrol1004:~# nova-manage project quota_usage_refresh --project etytree`

=== 2020-01-08 ===
* 10:53 arturo: icinga downtime all cloudvirts for 30 minutes to re-create all canary VMs&quot;

=== 2020-01-07 ===
* 11:12 arturo: icinga-downtime everything cloud* for 30 minutes to merge nova scheduler changes
* 10:02 arturo: icinga downtime cloudvirt1009 for 30 minutes to re-create canary VM ([[phab:T242078|T242078]])

=== 2020-01-06 ===
* 13:45 andrewbogott: restarting nova-api and nova-conductor on cloudcontrol1003 and 1004

=== 2020-01-04 ===
* 16:34 arturo: icinga downtime cloudvirt1024 for 2 months because hardware errors ([[phab:T241884|T241884]])

=== 2019-12-31 ===
* 11:46 andrewbogott: I couldn't!
* 11:40 andrewbogott: restarting cloudservices2002-dev to see if I can reproduce an issue I saw earlier

=== 2019-12-25 ===
* 10:13 arturo: icinga downtime for 30 minutes the whole cloud* lab* fleet to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/560575 (will restart some openstack components)

=== 2019-12-24 ===
* 15:13 arturo: icinga downtime all the lab* fleet for nova password change for 1h
* 14:39 arturo: icinga downtime all the cloud* fleet for nova password change for 1h

=== 2019-12-23 ===
* 11:13 arturo: enable puppet in cloudcontrol1003/1004
* 10:40 arturo: disable puppet in cloudcontrol1003/1004 while doing changes related to python-ldap

=== 2019-12-22 ===
* 23:48 andrewbogott: restarting nova-conductor and nova-api on cloudcontrol1003 and 1004
* 09:45 arturo: cloudvirt1013 is back (did it alone) [[phab:T241313|T241313]]
* 09:37 arturo: cloudvirt1013 is down for good. Apparently powered off. I can't even reach it via iLO

=== 2019-12-20 ===
* 12:43 arturo: icinga downtime cloudmetrics1001 for 128 hours

=== 2019-12-18 ===
* 12:55 arturo: [codfw1dev] created a new subnet neutron object to hold the new CIDR for floating IPs (cloud-codfw1dev-floating - 185.15.57.0/29) [[phab:T239347|T239347]]

=== 2019-12-17 ===
* 07:21 andrewbogott: deploying horizon/train to labweb1001/1002

=== 2019-12-12 ===
* 06:11 arturo: schedule 4h downtime for labstores
* 05:57 arturo: schedule 4h downtime for cloudvirts and other openstack components due to upgrade ops

=== 2019-12-02 ===
* 06:28 andrewbogott: running  nova-manage db sync on eqiad1
* 06:27 andrewbogott: running nova-manage cell_v2 map_cell0 on eqiad1

=== 2019-11-21 ===
* 16:07 jeh: created replica indexes and views for szywiki [[phab:T237373|T237373]]
* 15:48 jeh: creating replica indexes and views for shywiktionary [[phab:T238115|T238115]]
* 15:48 jeh: creating replica indexes and views for gcrwiki [[phab:T238114|T238114]]
* 15:46 jeh: creating replica indexes and views for minwiktionary [[phab:T238522|T238522]]
* 15:36 jeh: creating replica indexes and views for gewikimedia [[phab:T236404|T236404]]

=== 2019-11-18 ===
* 19:27 andrewbogott: repooling labsdb1011
* 18:54 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1011    [[phab:T238480|T238480]]
* 18:44 andrewbogott: depooling labsdb1011 and killing remaining user queries  [[phab:T238480|T238480]]
* 18:42 andrewbogott: repooled labsdb1009 and 1010  [[phab:T238480|T238480]]
* 18:19 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1010    [[phab:T238480|T238480]]
* 18:18 andrewbogott: depooling labsdb1010, killing remaining user queries
* 17:46 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1009    [[phab:T238480|T238480]]
* 17:38 andrewbogott: depooling labsdb1009, killing remaining user queries
* 16:54 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1012   [[phab:T237509|T237509]]

=== 2019-11-15 ===
* 20:04 andrewbogott: repool labdb1011 ([[phab:T237509|T237509]])
* 19:29 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1011
* 19:25 andrewbogott: depooling labsdb1011, killing remaining queries
* 19:25 andrewbogott: repooling labsdb1010
* 18:59 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1012
* 18:57 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1010
* 18:54 andrewbogott: depooling labsdb1010, killing remaining user queries
* 18:54 andrewbogott: depooled labsdb1009, ran maintain-views —clean —all-databases —replace-all, repooled

=== 2019-11-11 ===
* 13:10 arturo: cloudweb2001-dev: disable puppet and redirect stderr in the loadExitNodes.php cron script to prevent cronspam while we investigate the cause of the issue ([[phab:T237971|T237971]])

=== 2019-11-05 ===
* 11:59 arturo: icinga downtime for 1h cloudcontrol1004, cloudnet1003, cloudvirt1017/1020/1022 for PDU operations in the rack [[phab:T227542|T227542]]

=== 2019-11-04 ===
* 21:55 andrewbogott: deleting a ton of wikitech hiera pages that were either no-ops or refer to nonexistent VMs or prefixes

=== 2019-10-31 ===
* 11:01 arturo: icinga-downtimed cloudvirt1030 and cloudservices1003 for 1h due to PDU upgrade operations [[phab:T227543|T227543]]

=== 2019-10-30 ===
* 22:43 jeh: reboot cloud-bootstrapvz-stretch to resolve bad bootstrapvz build

=== 2019-10-29 ===
* 10:52 arturo: icinga downtime cloudvirt1001/1002/1024/1018/1012/1009/1015/1008 for 1h [[phab:T227538|T227538]]

=== 2019-10-25 ===
* 10:45 arturo: icinga downtime toolschecker for 1 to upgrade clouddb1002 mariadb (toolsdb secondary) ([[phab:T236384|T236384]] , [[phab:T236420|T236420]])

=== 2019-10-24 ===
* 12:30 arturo: starting cloudvirt1019, PDU operations ended ([[phab:T227540|T227540]])
* 11:58 arturo: icinga downtime for 2h ([[phab:T227540|T227540]]) cloudvirt1019
* 11:15 arturo: poweroff cloudvirt1019 during the PDU operations ([[phab:T227540|T227540]])
* 11:10 arturo: icinga downtime for 2h ([[phab:T227540|T227540]]) toolschecker
* 10:58 arturo: icinga downtime for 1h ([[phab:T227540|T227540]]) cloudvirt100[3-7], cloudvirt1019, cloudvirt1016, cloudvirt1021, cloudvirt1013, cloudnet1004

=== 2019-10-23 ===
* 09:23 arturo: cloudvirt1026 reboot ended OK
* 09:12 arturo: rebooting cloudvirt1026 for kernel upgrade
* 09:09 arturo: cloudvirt1025 reboot ended OK
* 09:00 arturo: rebooting cloudvirt1025 for kernel upgrade
* 08:51 arturo: icinga downtime cloudvirt1025/1026 for reboots

=== 2019-10-18 ===
* 16:01 arturo: created the `eqiad1.wikimedia.cloud` DNS zone ([[phab:T235846|T235846]])
* 14:27 andrewbogott: deleted a bunch of leaked VMS from earlier today from the admin-monitoring project.   Fullstack leaks due to an api outage, maybe?
* 10:44 arturo: double max_message_size from 40KB to 80KB in the cloud-admin mailing list. A simple email with a couple of quotes can go over the 40KB limit.

=== 2019-10-16 ===
* 21:59 jeh: resync wiki replica tool and user accounts [[phab:T235697|T235697]]
* 09:40 arturo: reboot of cloudvirt1030 went fine
* 09:28 arturo: reboot of cloudvirt1029 went fine
* 09:28 arturo: rebooting cloudvirt1030 for kernel updates
* 09:12 arturo: rebooting cloudvirt1029 for kernel updates
* 09:11 arturo: reboot of cloudvirt1028 went fine
* 09:00 arturo: rebooting cloudvirt1028 for kernel updates
* 08:56 arturo: icinga downtime cloudvirt[1028-1030].eqiad.wmnet for 1h for reboots

=== 2019-10-15 ===
* 13:30 jeh: creating indexes and views for banwiki [[phab:T234770|T234770]]

=== 2019-10-10 ===
* 18:55 bd808: Created indexes and views for nqowiki ([[phab:T230543|T230543]])
* 11:59 arturo: network switch hardware is down affecting cloudvirt1025/1026 ([[phab:T227536|T227536]]) VMs are supposed to be online but unreachable

=== 2019-10-09 ===
* 10:44 arturo: cloudvirt1013 rebooted well
* 10:32 arturo: cloudvirt1013 is rebooting
* 10:32 arturo: cloudvirt1012 rebooted just fine (very slow, 35 VMs)
* 10:21 arturo: cloudvirt1012 is rebooting
* 10:19 arturo: cloudvirt1009 rebooted just fine (very slow though)
* 10:07 arturo: cloudvirt1009 is rebooting
* 10:06 arturo: cloudvirt1008 rebooted just fine (very slow though)
* 09:58 arturo: cloudvirt1008 is rebooting
* 09:52 arturo: icinga downtime toolschecker, paws, etc for 2h, because cloudvirt reboots

=== 2019-10-07 ===
* 14:07 arturo: horizon is disabled for maintenance ([[phab:T212302|T212302]])
* 14:00 arturo: starting scheduled maintenance: upgrading eqiad1 from openstack mitaka to newton

=== 2019-10-02 ===
* 15:23 arturo: codfw1dev renaming net/subnet objects to a more modern naming scheme [[phab:T233665|T233665]]
* 12:49 arturo: codfw1dev delete all floating ip allocations in the deployment for mangling the network config for testing [[phab:T233665|T233665]]
* 12:47 arturo: codfw1dev deleting all VMs in the deployment for mangling the network config for testing [[phab:T233665|T233665]]
* 11:08 arturo: codfw1dev rebooting cloudnet2002-dev and cloudnet2003-dev for testing [[phab:T233665|T233665]]
* 10:31 arturo: codfw1dev: add cloudinstances2b-gw router to the l3 agent in cloudnet2003-dev
* 09:59 arturo: codfw1dev: cleanup leftover &quot;HA port tenant admin&quot; in neutron (ports from missing servers)
* 09:46 arturo: codfw1dev: cleanup leftover neutron agents

=== 2019-09-30 ===
* 10:21 arturo: we installed ferm in every VM by mistake. Deleting it and forcing a puppet agent run to try to go back to a clean state.
* 09:38 arturo: downtime toolschecker for 24h
* 09:33 arturo: force update ferm cloud-wide (in all VMs) for [[phab:T153468|T153468]]

=== 2019-08-18 ===
* 10:39 arturo: rebooting cloudvirt1023 for new interface names configuration
* 10:34 arturo: downtimed cloudvirt1023 for 2 days

=== 2019-08-05 ===
* 17:17 bd808: Set downtime on gridengine and kubernetes webservice checks in icinga until 2019-09-02 (flaky tests)

=== 2019-07-29 ===
* 20:14 bd808: Restarted maintain-kubeusers on tools-k8s-master-01 ([[phab:T194859|T194859]])

=== 2019-07-25 ===
* 12:32 arturo: eqiad1/glance: debian-9.9-stretch image deprecates debian-9.8-stretch ([[phab:T228983|T228983]])
* 09:59 arturo: (codfw1dev) drop missing glance images ([[phab:T228972|T228972]])
* 09:32 arturo: (codfw1dev) deleting a bunch of VMs that were running in now missing hypervisors
* 09:31 arturo: (codfw1dev) deleting a bunch of VMs in ERROR and SHUTDOWN state
* 09:27 arturo: last log entry refers to the codfw1dev deployment
* 09:27 arturo: cleanup `nova service-list` from old hypervisors (labtest*)
* 09:23 arturo: refreshed nova DB grants in clouddb2001-dev for the codfw1dev deployment
* 08:47 arturo: cleanup the cloud-announce pending emails (spam)

=== 2019-07-23 ===
* 19:43 andrewbogott: restarting rabbitmq-server on cloudcontrol1003 and 1004

=== 2019-07-22 ===
* 23:44 bd808: Restarted maintain-kubeusers on tools-k8s-master-01 ([[phab:T228529|T228529]])

=== 2019-07-11 ===
* 22:07 bd808: Ran `sudo systemctl stop designate_floating_ip_ptr_records_updater.service` on cloudcontrol1003
* 22:01 bd808: `sudo apt-get install python2.7-dbg` on cloudcontrol1003 to debug hung python process
* 21:48 bd808: Ran `sudo systemctl stop designate_floating_ip_ptr_records_updater.service` on cloudcontrol1004

=== 2019-06-25 ===
* 16:05 bstorm_: updated python3.4 to update4 wherever it was installed on Jessie VMs to prevent issues with broken update3.
* 14:56 bstorm_: Updated python 3.4 on the labs-puppetmaster server

=== 2019-06-03 ===
* 15:55 arturo: [[phab:T221769|T221769]] rebooting cloudservices1003 after bootstrapping is apparently completed

=== 2019-05-28 ===
* 21:42 bstorm_: unmounting labstore1003-scratch on all cloud clients
* 18:14 bstorm_: [[phab:T209527|T209527]] switched mounts from labstore1003 to cloudstore1008 for scratch

=== 2019-05-20 ===
* 17:25 arturo: [[phab:T223923|T223923]] dropped compat-network config from /etc/network/interfaces in eqiad1/codfw1dev neutron nodes
* 17:22 arturo: [[phab:T223923|T223923]] dropped br-compat bridges and vlan interfaces (1102 and 2102) in eqiad1/codfw1dev neutron nodes
* 17:07 arturo: [[phab:T223923|T223923]] dropped compat-network configuration from the neutron database in eqiad1
* 16:55 arturo: [[phab:T223923|T223923]] dropped compat-network configuration from the neutron database in codfw1dev

=== 2019-05-15 ===
* 17:00 andrewbogott: touching /root/firstboot_done on all VMs that cumin can reach.  This will prevent firstboot.sh from running a second time if/when any of these are rebooted.  [[phab:T223370|T223370]]

=== 2019-04-26 ===
* 15:51 arturo: andrew updated dns servers for the cloud-instances2-b-eqiad subnet in neutron: 208.80.154.143 and 208.80.154.24

=== 2019-04-25 ===
* 11:14 arturo: [[phab:T221760|T221760]] increased size of conntrack table

=== 2019-04-24 ===
* 12:54 arturo: [[phab:T220051|T220051]] puppet broken in every VM in Cloud VPS, fixing right now

=== 2019-04-22 ===
* 11:14 arturo: create by hand /var/cache/labsaliaser/labs-ip-aliases.json in cloudservices2002-dev ([[phab:T218575|T218575]])

=== 2019-04-16 ===
* 22:55 bd808: cloudcontrol2003-dev: added `exit 0` to /etc/cron.hourly/keystone to stop cron spam on partially configured cluster
* 12:08 arturo: rebooting cloudvirt200[123]-dev because deep changes in config
* 11:27 arturo: [[phab:T219626|T219626]] add DB grants for neutron and glnace to clouddb2001-dev (codfw1dev)
* 10:37 arturo: [[phab:T219626|T219626]] replace 208.80.153.75 with 208.80.153.59 in the clouddb2001-dev database (codfw1dev deployment)
* 10:30 arturo: [[phab:T219626|T219626]] replace labtestcontrol2003 with cloudcontrol2001-dev in the clouddb2001-dev database (codfw1dev deployment)

=== 2019-04-15 ===
* 13:08 arturo: [[phab:T219626|T219626]] add DB grants for keystone/nova/nova_api to clouddb2001-dev (codfw1dev)

=== 2019-04-13 ===
* 18:25 bd808: Restarted nova-compute service on cloudvirt1015 ([[phab:T220853|T220853]])

=== 2019-04-11 ===
* 12:00 arturo: [[phab:T151704|T151704]] deploying oidentd to cloudnet1xxx servers

=== 2019-04-02 ===
* 19:52 andrewbogott: installed new base Stretch image.  Updated packages, and runs apt-get dist-upgrade on first boot.

=== 2019-03-29 ===
* 14:34 andrewbogott: moving tools-static.wmflabs.org to point to tools-static-13 in eqiad1-r
* 00:00 bstorm_: [[phab:T193264|T193264]] Added osm.db.svc.eqiad.wmflabs to cloud DNS

=== 2019-03-25 ===
* 00:40 bd808: Restarted maintain-dbusers on labstore1004. Process hung up on failed LDAP connection.

=== 2019-03-21 ===
* 19:32 andrewbogott: restarting keystone on cloudcontrol1003

=== 2019-03-15 ===
* 16:00 gtirloni: increased nscd cache size ([[phab:T217280|T217280]])

=== 2019-03-14 ===
* 19:04 gtirloni: bstorm started nfsd on labstore1006 ([[phab:T218341|T218341]])
* 16:42 gtirloni: published new debian-9.8 image ([[phab:T218314|T218314]])

=== 2019-03-04 ===
* 19:37 bstorm_: umounted /mnt/nfs/dumps-labstore1006.wikimedia.org across all VPS projects for [[phab:T217473|T217473]]

=== 2019-02-26 ===
* 12:46 gtirloni: shutdown toolsbeta-sgegrid-master (cronspam)

=== 2019-02-25 ===
* 10:32 gtirloni: restarted nfsd on labstore1004

=== 2019-02-21 ===
* 09:09 gtirloni: restarted uwsgi-labspuppetbackend.service on labpuppetmaster1001
* 07:42 gtirloni: created project cloudstore
* 07:36 gtirloni: deleted wmcs-nfs project

=== 2019-02-20 ===
* 21:58 andrewbogott: silencing shinken and disabling puppet on shinken-02 for now

=== 2019-02-19 ===
* 12:00 gtirloni: added nagios@icinga2001.wikimedia.org to cloud-admin-feed@ allowed senders

=== 2019-02-18 ===
* 20:21 gtirloni: downtimed cloudvirt1020
* 20:12 gtirloni: ran `labs-ip-alias-dump.py` on cloudservices/labservices servers

=== 2019-02-15 ===
* 13:10 arturo: [[phab:T216239|T216239]] labvirt1019 has been drained
* 12:22 arturo: [[phab:T216239|T216239]] draining labvirt1009 with a command like this: `root@cloudcontrol1004:~# wmcs-cold-migrate --region eqiad --nova-db nova 2c0cf363-c7c3-42ad-94bd-{{Gerrit|e586f2492321}} labvirt1001`
* 12:02 arturo: more nova service cleanups in the database (labvirts that were reallocated to eqiad1)
* 11:34 arturo: [[phab:T216190|T216190]] cleanup from nova database `nova service-delete 35`
* 03:50 andrewbogott: updated VPS base images for Jessie and Stretch, now featuring Stretch 9.7

=== 2019-02-11 ===
* 18:13 gtirloni: cleaned old metrics data in labmon1001 [[phab:T215417|T215417]]
* 15:28 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1011
* 14:18 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1010

=== 2019-02-08 ===
* 14:56 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1009

=== 2019-02-06 ===
* 11:47 gtirloni: downtimed labmon100{1,2} [[phab:T215399|T215399]]
* 00:17 bstorm_: [[phab:T214106|T214106]] deleted bstorm-test2 project to clean up

=== 2019-02-05 ===
* 10:48 arturo: labmon1001 is now part of the 'eqiad1-r' region

=== 2019-02-01 ===
* 09:54 arturo: moving canary1015-01 VM instance from cloudvirt1024 back to cloudvirt1015

=== 2019-01-31 ===
* 12:44 arturo: [[phab:T215012|T215012]] depooling cloudvirt1015 and migrating all VMs to cloudvirt1024

=== 2019-01-25 ===
* 20:11 gtirloni: deleted project yandex-proxy [[phab:T212306|T212306]]
* 20:11 gtirloni: deleted project [[phab:T212306|T212306]]

=== 2019-01-24 ===
* 11:50 arturo: [[phab:T213925|T213925]] modify subnet cloud-instances-transport1-b-eqiad1 to avoid floating IP allocations from here
* 11:07 arturo: [[phab:T214299|T214299]] failover cloudnet1003 to cloudnet1004
* 10:03 arturo: [[phab:T214299|T214299]] reimage cloudnet1004 to debian stretch
* 09:51 arturo: [[phab:T214299|T214299]] failover cloudnet1004 to cloudnet1003

=== 2019-01-22 ===
* 19:19 arturo: [[phab:T214299|T214299]] stretch cloudnet1003 is apparently all set
* 18:40 arturo: [[phab:T214299|T214299]] manually delete from neutron agents from cloudnet1003 (must be added again after reimage, with new uuids)
* 18:37 arturo: [[phab:T214299|T214299]] reimaging cloudnet1003 as debian stretch
* 17:35 jbond42: starting roll out of apt package updates to
* 14:41 gtirloni: [[phab:T214369|T214369]] deployed new jessie and stretch VM images

=== 2019-01-21 ===
* 18:29 gtirloni: installed libguestfs-tools on cloudvirt1021

=== 2019-01-16 ===
* 14:21 andrewbogott: stopping old VPS proxies in eqiad — [[phab:T213540|T213540]]

=== 2019-01-15 ===
* 14:20 andrewbogott: changing tools.wmflabs.org to point to tools-proxy-03 in eqiad1

=== 2019-01-13 ===
* 20:00 andrewbogott: VPS proxies are now running in eqiad1 on proxy-01.  Old VMs will wait a bit for deletion.  [[phab:T213540|T213540]]
* 19:12 andrewbogott: moving the VPS proxy API backend to proxy-01.project-proxy.eqiad.wmflabs, as per [[phab:T213540|T213540]]
* 17:11 andrewbogott: moving all VPS dynamic proxies to proxy-eqiad1.wmflabs.org aka proxy-01.project-proxy.eqiad.wmflabs, as per [[phab:T213540|T213540]]

=== 2019-01-09 ===
* 22:21 bd808: neutron quota-update --tenant-id tools --port 256

=== 2019-01-08 ===
* 18:59 bd808: Definately did NOT delete uid=novaadmin,ou=people,dc=wikimedia,dc=org
* 18:59 bd808: Deleted LDAP user uid=neutron,ou=people,dc=wikimedia,dc=org
* 18:58 bd808: Deleted LDAP user uid=novaadmin,ou=people,dc=wikimedia,dc=org

=== 2019-01-06 ===
* 22:03 bd808: Set floatingip quota of 60 for tools project in eqiad1-r region ([[phab:T212360|T212360]])

=== 2018-12-20 ===
* 17:10 arturo: [[phab:T207663|T207663]] renumbered transport network in eqiad1

=== 2018-12-05 ===
* 17:59 arturo: [[phab:T207663|T207663]] changed labtestn transport network addressing from private to public

=== 2018-12-03 ===
* 13:25 arturo: [[phab:T202886|T202886]] create again PTR records after dnsleak.py fix

=== 2018-11-30 ===
* 14:08 arturo: running dns leaks cleanup `root@cloudcontrol1003:~# /root/novastats/dnsleaks.py --delete`

=== 2018-11-28 ===
* 17:33 gtirloni: deleted contintcloud project ([[phab:T209644|T209644]])

=== 2018-11-27 ===
* 13:32 gtirloni: enabled DRBD stats collection on labstore100[4-5] [[phab:T208446|T208446]]

=== 2018-11-22 ===
* 07:12 gtirloni: deployed new debian-9.6-stretch image

=== 2018-11-21 ===
* 10:48 arturo: re-created compat-net as not shared in labtestn to test stuff related to [[phab:T209954|T209954]]

=== 2018-11-16 ===
* 12:43 gtirloni: armed keyholder on labpuppetmaster1001/1002 after reboots
* 12:08 gtirloni: rebooted labpuppetmaster1001 ([[phab:T207377|T207377]])
* 11:57 gtirloni: rebooted labpuppetmaster1002 ([[phab:T207377|T207377]])

=== 2018-11-14 ===
* 17:19 gtirloni: added cloudvirt1016 to scheduler pool ([[phab:T209426|T209426]])
* 15:41 gtirloni: reimaging labvirt1016 as cloudvirt1016
* 15:14 gtirloni: reset-failed systemd unit nova-scheduler on cloudcontrol1004
* 13:52 gtirloni: rebooted labservices1002 after package upgrades ([[phab:T207377|T207377]])
* 13:23 gtirloni: rebooted labstore2004 after package upgrades ([[phab:T207377|T207377]])
* 13:20 gtirloni: rebooted labstore2003 after package upgrades ([[phab:T207377|T207377]])
* 13:20 gtirloni: rebooted labstore2001/labstore2003 after package upgrades ([[phab:T207377|T207377]])
* 12:08 gtirloni: rebooted labnet1002 after package upgrades
* 12:01 gtirloni: rebooted labmon1002 after package upgrades
* 11:41 gtirloni: rebooted labcontrol1002 after package upgrades
* 11:15 gtirloni: rebooted cloudcontrol1004 after package upgrades

=== 2018-11-09 ===
* 18:17 gtirloni: restarted neutron-linuxbridge-agent on cloudvirt1018/1023

=== 2018-11-08 ===
* 11:00 gtirloni: Added novaproxy-02 to $CACHES
* 10:50 gtirloni: Added cloudvirt1017 to eqiad1 region

=== 2018-11-07 ===
* 13:49 arturo: [[phab:T208733|T208733]] moving labvirt1017 from main deployment to eqiad1 and renaming it to cloudvirt1017

=== 2018-10-22 ===
* 16:24 arturo: [[phab:T206261|T206261]] another update to dmz_cidr in eqiad1
* 10:26 arturo: change again in dmz_cidr in eqiad1: VMs will connect between them without NAT even when using floating IPs ([[phab:T206261|T206261]])

=== 2018-10-19 ===
* 12:02 arturo: revert change in dmz_cidr in eqiad1 for now ([[phab:T206261|T206261]])
* 11:16 arturo: change in dmz_cidr in eqiad1: VMs will connect between them without NAT even when using floating IPs ([[phab:T206261|T206261]])
* 10:14 arturo: we have new virt servers in the eqiad1 deployment since past week and this week: cloudvirt1018, cloudvirt1023, cloudvirt1024

=== 2018-09-26 ===
* 10:40 arturo: [[phab:T205524|T205524]] all sorts of restarts in all neutron daemons
* 10:20 arturo: [[phab:T205524|T205524]] stop/start all neutron agents in cloudnet1003.eqiad.wmnet
* 10:13 arturo: [[phab:T205524|T205524]] restart all agents in cloudnet1004.eqiad.wmnet
* 10:10 arturo: restart neutron-server in cloudcontrol1003, investigating [[phab:T205524|T205524]]

=== 2018-09-24 ===
* 10:57 arturo: try to increase floating ip allocation pool in eqiad1. Of 185.15.56.0/25 we are using only 185.15.56.10-185.15.56.31, I don't know why. Let's use 185.15.56.2-185.15.56.126

=== 2018-09-21 ===
* 17:18 bd808: Running `sudo maintain-meta_p --all-databases --purge` across labsdb10(09{{!}}10{{!}}11) for [[phab:T201890|T201890]]

=== 2018-09-17 ===
* 22:08 bd808: Granted gtirloni project roles of admin, projectadmin, and user

=== 2018-09-12 ===
* 11:20 arturo: [[phab:T202636|T202636]] distributing default routes using classless-static-route for all VMs in main/labtest (dnsmasq/nova-network)

=== 2018-09-11 ===
* 16:52 arturo: again, restarted nova-network after killing all dnsmasq procs in labnet1001 for [[phab:T202636|T202636]]
* 16:08 arturo: restarted nova-network after killing all dnsmasq procs in labnet1001 for [[phab:T202636|T202636]]
* 10:53 arturo: [[phab:T202636|T202636]] creating all the compat-network configuration in neutron
* 10:36 arturo: [[phab:T202636|T202636]] creating br-compat bridge in eqiad1 for the compat network
* 10:33 arturo: [[phab:T202636|T202636]] manually reserve 10.68.23.253 (in nova-network)

=== 2018-09-10 ===
* 22:46 andrewbogott: deleting all VMs on labvirt1019 and 1020 as prep for [[phab:T204003|T204003]]

=== 2018-08-30 ===
* 15:46 andrewbogott: restarting rabbitmq-server on cloudcontrol1003
* 13:07 arturo: [[phab:T202636|T202636]] internal network routing now exists in labtest/labtestn for VM to communicate with each other

=== 2018-08-28 ===
* 11:04 arturo: [[phab:T202549|T202549]] eqiad1 databases are all now running in m5-master. Mysql has been cleaned from cloudcontrol100[3,4]

=== 2018-08-23 ===
* 16:17 arturo: [[phab:T188589|T188589]] bstorm_ merged patch to reduce nova DB connection usage
* 13:15 arturo: [[phab:T202115|T202115]] `root@cloudcontrol1003:~# neutron subnet-update --allocation-pool start=10.64.22.4,end=10.64.22.4 e4fb2771-a361-4add-ac4e-280cc300c59f`
* 13:10 arturo: [[phab:T202115|T202115]] (was `{&quot;start&quot;: &quot;10.64.22.2&quot;, &quot;end&quot;: &quot;10.64.22.254&quot;}` )
* 13:08 arturo: [[phab:T202115|T202115]] `root@cloudcontrol1003:~# neutron subnet-update --allocation-pool start=10.64.22.254,end=10.64.22.254 e4fb2771-a361-4add-ac4e-280cc300c59f`

=== 2018-08-22 ===
* 15:28 arturo: cleanup local glance,keystone databases in cloudcontrol1003.wikimedia.org (already in m5-master)
* 15:27 arturo: cleanup local keystone database in cloudcontrol1003.wikimedia.org (already in m5-master)

=== 2018-08-21 ===
* 15:39 andrewbogott: initial test message
* 10:31 arturo: eqiad1 remove leftover port for HA on labnet1004
* 10:15 arturo: test

=== 2018-05-07 ===
* 18:07 bstorm_: stopped the toolhistory job because it is totally broken and fills /tmp.

=== 2018-02-09 ===
* 00:55 bd808: Added Arturo Borrero Gonzalez and Bstorm as project members
* 00:54 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])

{{SAL|Project Name=admin}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>6li6xrsabi60h8xwjcdcika14f87f6u</sha1>
    </revision>
    <revision>
      <id>2309617</id>
      <parentid>2309616</parentid>
      <timestamp>2025-06-07T19:08:04Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services</comment>
      <origin>2309617</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1268764" sha1="ckvspp2hhoqjvh322rldyvuvoqj7zbw" xml:space="preserve">=== 2025-06-07 ===
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova

=== 2025-06-06 ===
* 20:59 andrewbogott: restarting all designate services on all cloudcontrols in eqiad1

=== 2025-06-05 ===
* 20:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,octavia
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,octavia
* 15:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:52 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2025-06-03 ===
* 22:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 22:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244
* 20:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:44 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:43 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 17:50 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T395910|T395910]])
* 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T395910|T395910]])
* 17:36 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-06-02 ===
* 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 21:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237
* 21:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242
* 17:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241
* 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240
* 15:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239
* 00:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T394333|T394333]])

=== 2025-06-01 ===
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T394333|T394333]])

=== 2025-05-31 ===
* 23:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services ([[phab:T395742|T395742]])
* 23:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services ([[phab:T395742|T395742]])
* 23:38 andrewbogott: failing over from cloudnet1005 to 1006 in hopes of unsticking [[phab:T395742|T395742]]
* 23:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,neutron ([[phab:T395742|T395742]])
* 23:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,neutron ([[phab:T395742|T395742]])

=== 2025-05-30 ===
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-28 ===
* 22:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1076.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1076.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1075.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1075.eqiad.wmnet' ([[phab:T390914|T390914]])
* 22:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1074.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1074.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1073.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1073.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1072.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1072.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1071.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1071.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1070.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1070.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1069.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1069.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1068.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1068.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T390914|T390914]])
* 21:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T390914|T390914]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudrabbot1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbot1003.eqiad.wmnet' ([[phab:T390914|T390914]])
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 15:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,designate
* 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate

=== 2025-05-26 ===
* 13:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet2006-dev.codfw.wmnet
* 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet2006-dev.codfw.wmnet
* 13:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet2005-dev.codfw.wmnet
* 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet2005-dev.codfw.wmnet
* 12:54 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=99)
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.roll_reboot_cloudnets
* 12:49 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=99)
* 12:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.roll_reboot_cloudnets
* 12:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::control&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::control&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::codfw1dev::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 07:52 taavi: add new /dev/sdb back to software raid after it was replaced in cloudcephmon1004

=== 2025-05-23 ===
* 09:00 dhinus: failover dumps_dist_active_vps to clouddumps1001 ([[phab:T383723|T383723]])

=== 2025-05-20 ===
* 19:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 15:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 14:57 andrewbogott: resetting eqiad1 rabbitmq in an attempt to resolve [[phab:T394790|T394790]]
* 03:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 03:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 02:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 00:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 00:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])

=== 2025-05-19 ===
* 22:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 22:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 22:43 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for service: project,nova
* 22:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova
* 22:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:26 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for service: project,nova
* 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova
* 22:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 21:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 19:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 18:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 18:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T394727|T394727]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T394727|T394727]])
* 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1075.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1075.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1072.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1072.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1076.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1076.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1073.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1073.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-15 ===
* 15:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/235
* 14:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/234
* 13:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-05-14 ===
* 17:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 17:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/231
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/231
* 12:59 taavi: powercycle unresponsive cloudnet2006-dev

=== 2025-05-12 ===
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 08:17 taavi: powercycle clouservices2005-dev.codfw.wmnet
* 03:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 03:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 02:36 andrewbogott: rebooting cloudnet2005-dev from mgmt  -- ssh is failing and the console shows a user prompt but not a password prompt.

=== 2025-05-11 ===
* 13:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova

=== 2025-05-07 ===
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet' ([[phab:T390914|T390914]])
* 20:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 20:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 19:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.codfw.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.eqiad.wmnet' ([[phab:T390914|T390914]])
* 18:19 andrewbogott: upgrading codfw1dev to version 'epoxy' [[phab:T390914|T390914]]
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T390914|T390914]])
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T390914|T390914]])
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum,heat
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum,heat
* 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum,heat
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum,heat
* 16:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum
* 16:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,nova
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 15:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,nova
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova
* 13:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,neutron
* 13:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,neutron
* 13:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,neutron
* 13:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,neutron
* 12:48 taavi: updating all security group rules referencing old 172.16.0.0/21 subnet to reference new ip space instead ([[phab:T379175|T379175]])
* 10:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-05-06 ===
* 03:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)

=== 2025-05-05 ===
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 03:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 00:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 00:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 00:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 00:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-05-04 ===
* 23:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:53 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:41 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 22:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2025-05-03 ===
* 02:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])

=== 2025-05-02 ===
* 23:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 21:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T393196|T393196]])
* 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 16:20 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=97) ([[phab:T393196|T393196]])
* 16:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T393196|T393196]])
* 16:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 16:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrewbogott: sudo cumin --force O&lt;nowiki&gt;{&lt;/nowiki&gt;*&lt;nowiki&gt;}&lt;/nowiki&gt; &quot;dpkg --list {{!}} grep puppetserver &amp;&amp; systemctl restart puppetserver.service&quot; # work around a package update that is causing some puppetservers to error out

=== 2025-04-29 ===
* 13:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225
* 12:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225
* 11:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:55 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224
* 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223
* 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223
* 07:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 07:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221

=== 2025-04-28 ===
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
* 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:21 taavi: migrating opentofu managed default security group rules [[phab:T392799|T392799]]
* 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-26 ===
* 07:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0) ([[phab:T390134|T390134]])
* 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance ([[phab:T390134|T390134]])
* 07:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T390134|T390134]])
* 07:47 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T390134|T390134]])

=== 2025-04-25 ===
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218
* 12:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 04:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T390134|T390134]])

=== 2025-04-24 ===
* 17:25 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T390134|T390134]])
* 13:29 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for service: project,designate
* 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate

=== 2025-04-23 ===
* 21:45 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 21:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:45 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 21:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 21:21 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 19:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:03 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:32 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:29 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 16:29 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:17 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:27 arturo: enabling IPv6 dualstack on neutron virtual router ([[phab:T380174|T380174]])
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:14 arturo: enable IPv6 on cloudgw ([[phab:T380174|T380174]]) -- includes server reboot

=== 2025-04-22 ===
* 04:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)

=== 2025-04-21 ===
* 23:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 23:34 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 23:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 23:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 22:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:09 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 22:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 16:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 01:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 01:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-16 ===
* 16:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,cinder
* 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,cinder
* 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 arturo: merging network change in neutron https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/198
* 10:10 dcaro: upgrade spicerack on cloudcumin2001 to 10.1.0
* 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 10:07 dcaro: upgrade spicerack on cloudcumin1001 to 10.1.0

=== 2025-04-15 ===
* 15:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:38 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:31 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:30 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-14 ===
* 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 17:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:50 andrewbogott: granting 'tofuadmin' user inherited 'member' role in all projects, this should fix some policy mishaps
* 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-13 ===
* 22:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 22:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-11 ===
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-10 ===
* 12:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:05 arturo: [codfw1dev] root@cloudcontrol2004-dev:~# wmcs-makedomain --project testlabs --domain testlabs.codfw1dev.wmcloud.org --orig-project cloudinfra-codfw1dev ([[phab:T391325|T391325]])

=== 2025-04-09 ===
* 23:23 bd808: Rebooting tools-sgebastion-10 (login-buster.toolforge.org) for high load/unresponsive NFS mounts ([[phab:T391538|T391538]])
* 10:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 08:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 04:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 03:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:30 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 03:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 03:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 03:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services

=== 2025-04-08 ===
* 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet'
* 22:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet'
* 22:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1011.eqiad.wmnet'
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet'
* 11:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2025-04-07 ===
* 23:43 bd808: `sudo service maintain-dbusers restart` on cloudcontrol1007 after reports of missing replica.my.cnf and finding the journal for the service empty.
* 15:30 arturo: [codfw1dev] testlabs create a bunch of VMs by hand, like `networktests-vlan-legacy-floating` [[phab:T380728|T380728]]
* 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:23 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-04-06 ===
* 02:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 02:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-04 ===
* 15:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 05:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 05:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:10 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for all services
* 05:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:04 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for all services
* 05:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services

=== 2025-04-03 ===
* 22:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 21:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 20:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 20:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 20:11 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment eqiad1 for all services
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for all services
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 19:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1002.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudrabbit1001.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:29 taavi: run wmcs-wikireplica-dns to create transitional x3 CNAMEs [[phab:T390954|T390954]]

=== 2025-04-02 ===
* 20:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T381499|T381499]])
* 20:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T381499|T381499]])
* 19:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup2004.codfw.wmnet' ([[phab:T381499|T381499]])
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1004.eqiad.wmnet' ([[phab:T381499|T381499]])
* 18:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1003.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T381499|T381499]])
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.unset_maintenance (exit_code=0) ({{Gerrit|1133432}})
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.unset_maintenance ({{Gerrit|1133432}})
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:22 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1007.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:26 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T381499|T381499]])
* 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T381499|T381499]])
* 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T381499|T381499]])
* 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T381499|T381499]])
* 09:53 dhinus: systemctl restart maintain-dbusers.service (attempting to fix a weird issue)

=== 2025-04-01 ===
* 21:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate
* 21:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 21:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 21:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 21:00 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-31 ===
* 15:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 15:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 15:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 15:53 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2025-03-27 ===
* 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 14:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 12:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T390134|T390134]])
* 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T390134|T390134]])

=== 2025-03-26 ===
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-25 ===
* 14:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-24 ===
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-03-22 ===
* 03:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,designate
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate

=== 2025-03-19 ===
* 14:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,designate
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate
* 12:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,task_id,no_dologmsg,cluster_name,all_services,nova,glance,keystone,cinder,neutron,trove,magnum,heat,swift,designate,filter_nodes
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,task_id,no_dologmsg,cluster_name,all_services,nova,glance,keystone,cinder,neutron,trove,magnum,heat,swift,designate,filter_nodes
* 06:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 06:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 06:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 06:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet'
* 06:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 06:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet'
* 06:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet'
* 06:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 06:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet'
* 06:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 06:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 05:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet'
* 05:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 05:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 05:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
* 05:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
* 05:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudnet2005-dev.codfw.wmnet'
* 05:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 05:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 05:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 05:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 05:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 04:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 04:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 04:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet'
* 04:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 04:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 04:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet'
* 04:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet'
* 04:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet'
* 04:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet'
* 04:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet'
* 04:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 04:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 03:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 02:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T381499|T381499]])
* 02:12 andrewbogott: upgrading codfw1dev to openstack 'dalmation'  [[phab:T381499|T381499]]
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T381499|T381499]])

=== 2025-03-14 ===
* 13:36 volans: installed cumin v5.1.1 on cloudcumin* hosts

=== 2025-03-05 ===
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
* 00:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
* 00:43 andrewbogott: restarting all openstack services in hopes of gettting dns unstuck
* 00:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services

=== 2025-03-04 ===
* 08:52 arturo: [[phab:T387828|T387828]] depooled galera on cloudcontrol1005

=== 2025-03-01 ===
* 19:35 andrewbogott: installed new bookworm base images in eqiad1
* 19:35 andrewbogott: installed new bookworm and bullseye base images in codfw1dev

=== 2025-02-28 ===
* 15:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-27 ===
* 15:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-24 ===
* 00:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 00:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for main branch
* 00:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2025-02-23 ===
* 21:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-21 ===
* 12:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-20 ===
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T386083|T386083]])
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T386083|T386083]])

=== 2025-02-19 ===
* 13:26 arturo: manual failover of cloudgw1004 to cloudgw1003 [[phab:T382356|T382356]]
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-18 ===
* 12:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-02-13 ===
* 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-02-11 ===
* 11:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T386083|T386083]])
* 11:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T386083|T386083]])
* 11:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047' ([[phab:T386083|T386083]])
* 11:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047' ([[phab:T386083|T386083]])

=== 2025-02-07 ===
* 13:52 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet1005.eqiad.wmnet ([[phab:T384946|T384946]])
* 13:48 root@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet1005.eqiad.wmnet ([[phab:T384946|T384946]])
* 02:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 01:48 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 01:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 00:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2025-02-06 ===
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:45 andrewbogott: cold-migrating all remaining VMs in [[phab:T385264|T385264]] except for 'integration' and 'tools' VMs

=== 2025-02-05 ===
* 19:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 19:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2025-02-04 ===
* 12:48 arturo: replacing cloudgw1002 with cloudgw1004 - [[phab:T382356|T382356]]
* 09:22 arturo: fleet-wide restart of puppetservers [[phab:T385553|T385553]]

=== 2025-02-03 ===
* 13:42 andrewbogott: rebooting proxy-04.project-proxy for the ceph OSD mishap

=== 2025-02-02 ===
* 17:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 17:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 16:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2025-01-31 ===
* 01:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2025-01-30 ===
* 21:39 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 21:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:33 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 13:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 13:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 12:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1031.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1031.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 03:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 02:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 02:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 00:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 00:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)

=== 2025-01-29 ===
* 23:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 23:43 andrewbogott: resetting rabbitmq in eqiad1 in hopes that it will resolve mysterious openstack misbehaviors
* 20:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 19:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2009-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2009-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2006-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1007.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2004-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1007.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol2005-dev.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1005.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1005.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T384946|T384946]])
* 12:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-28 ===
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:51 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 13:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 13:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 13:02 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T348643|T348643]])
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 00:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 00:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T348643|T348643]])
* 00:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2025-01-27 ===
* 21:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 21:52 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 21:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 14:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-24 ===
* 13:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node

=== 2025-01-23 ===
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 20:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 15:49 dhinus: cumin 'P:base::cloud_production' 'rm /var/lib/prometheus/node.d/kernel-panic.prom' [[phab:T382961|T382961]]
* 15:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2025-01-22 ===
* 14:52 dhinus: cloudcumin[12]001 upgrade spicerack from 8.15.2 to 9.1.0

=== 2025-01-21 ===
* 13:42 andrewbogott: migrating/rebooting VMs as per earlier email,  [[phab:T383583|T383583]]

=== 2025-01-20 ===
* 14:23 dhinus: cumin upgraded form 4.2.0 to 5.0.0 on cloudcumin[12]001. patch [[phab:T346453|T346453]] reapplied.

=== 2025-01-14 ===
* 21:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T383583|T383583]])
* 21:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T383583|T383583]])
* 20:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T383583|T383583]])
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T383583|T383583]])
* 20:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T383583|T383583]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T383583|T383583]])
* 18:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.roll_restart_osd_daemons (exit_code=0)

=== 2025-01-13 ===
* 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.roll_restart_osd_daemons

=== 2025-01-10 ===
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2025-01-09 ===
* 20:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2025-01-08 ===
* 14:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 13:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2025-01-03 ===
* 21:01 bd808: `sudo service maintain-dbusers restart` on cloudcontrol1005. Report of missing replica.my.cnf and journalctl output empty due to log rotation. ([[phab:T382962|T382962]])

=== 2024-12-18 ===
* 15:33 arturo: cloudgw failover for [[phab:T382220|T382220]]

=== 2024-12-04 ===
* 17:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 17:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 15:09 andrewbogott: rebooted cloudinfra-cloudvps-puppetserver-1, it's so busy that it's unresponsive
* 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T380893|T380893]])

=== 2024-11-27 ===
* 13:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet'
* 13:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'

=== 2024-11-26 ===
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T380731|T380731]])
* 15:16 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:15 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:15 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:15 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:14 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/145
* 15:13 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/145
* 15:12 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:12 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:31 dcaro: added cloudcephmon1004 to the ceph mon pool
* 12:41 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 12:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 05:40 andrewbogott: rebooting tools-sgebastion-10.tools.eqiad1.wikimedia.cloud to get NFS things remounted
* 04:56 andrewbogott: 'systemctl restart nfs-server' on tools-nfs-2.tools.eqiad1.wikimedia.cloud

=== 2024-11-25 ===
* 14:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:40 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 13:55 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:28 arturo: create IPv6 networks in eqiad1 ([[phab:T380174|T380174]]), then reverted because network outage
* 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-23 ===
* 14:58 taavi: removed broken records for re-created VMs in .eqiad.wmflabs zone

=== 2024-11-22 ===
* 14:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:35 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:48 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-20 ===
* 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2024-11-19 ===
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:43 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:43 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:37 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:35 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 11:39 arturo: [codfw1dev] performing rabbit full reset [[phab:T380208|T380208]]
* 10:26 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:24 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 10:24 arturo: [codfw1dev] restart rabbitmq and nova/neutron services for [[phab:T380208|T380208]]

=== 2024-11-16 ===
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 07:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 07:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-15 ===
* 19:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 17:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:10 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 17:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 17:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:30 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:29 arturo: [codfw1dev] restart rabbitmq and designate
* 16:29 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-14 ===
* 09:52 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:48 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-11-13 ===
* 13:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/120
* 13:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/120

=== 2024-11-11 ===
* 14:49 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2024-11-08 ===
* 16:47 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:45 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 16:42 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:42 arturo: [codfw1dev] restart all nova services and rabbitmq out of despair
* 16:42 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:45 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:44 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:33 aborrero@cloudcumin2001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 13:32 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:28 arturo: [codfw1dev] restart rabbitmq, openstack services logs show connection errors

=== 2024-11-07 ===
* 17:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 17:42 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-06 ===
* 13:09 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
* 11:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/116
* 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/116
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-11-05 ===
* 10:24 arturo: [codfw1dev] disable puppet and make changes for testing [[phab:T378192|T378192]]

=== 2024-11-04 ===
* 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:09 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:08 aborrero@cloudcumin2001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:08 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 14:08 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:08 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 13:52 arturo: [codfw1dev] live-hack cloudlb2001-dev and cloudcontrol2004-dev for [[phab:T378192|T378192]]
* 13:52 arturo: [codfw1dev] restart rabbitmq
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-29 ===
* 15:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-28 ===
* 17:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:00 arturo: [codfw1dev] restarting rabbitmq, misbehaving
* 17:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 dhinus: apt full-upgrade and reboot for cloudcumin*
* 16:21 dhinus: upgrade spicerack from 8.8.0 to 8.15.1 on cloudcumin*

=== 2024-10-24 ===
* 15:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-22 ===
* 15:02 taavi: recover access to User:Labslogbot [[phab:T376220|T376220]]

=== 2024-10-21 ===
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:06 arturo: [codfw1dev] restart rabbitmq, tofu shows error talking to the designate API

=== 2024-10-16 ===
* 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-15 ===
* 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:38 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:33 arturo: cloudgw maintenance, firewall change for [[phab:T374714|T374714]]
* 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-11 ===
* 15:31 arturo: cloudgw maintenance firewall change  [[phab:T374716|T374716]]
* 09:51 arturo: cloudgw network maintenance related to [[phab:T376879|T376879]]
* 08:33 arturo: [codfw1dev] reboot cloudgw2002-dev/2003-dev because network connectivity issues

=== 2024-10-10 ===
* 12:04 arturo: manual network failover in cloudgw because maintenance related to [[phab:T376879|T376879]]
* 10:40 dhinus: cumin 'cloudrabbit*' 'systemctl restart rabbitmq-server' [[phab:T376802|T376802]]
* 09:19 arturo: [codfw1dev] enable IPv6 on cloudgw ([[phab:T374716|T374716]])

=== 2024-10-09 ===
* 14:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:20 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:19 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:19 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:16 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 14:16 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:58 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:58 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:55 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:55 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:54 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:53 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:53 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
* 10:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95
* 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95
* 10:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 10:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

=== 2024-10-08 ===
* 11:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-07 ===
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-04 ===
* 15:12 arturo: cloudservice1005/1006: enable puppet and restore /etc/powerdns/recursor.conf to non-debug mode ([[phab:T374830|T374830]])
* 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:32 arturo: cloudservice1005/1006: disable puppet and set `quiet=no` in /etc/powerdns/recursor.conf ([[phab:T374830|T374830]])

=== 2024-10-03 ===
* 14:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:54 arturo: [codfw1dev] delete default security group rule list, now tracking them via tofu-infra
* 14:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:51 arturo: delete default security group rule list, now tracking them via tofu-infra
* 14:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-10-02 ===
* 15:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:21 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:16 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:15 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:05 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 10:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch ([[phab:T376211|T376211]])
* 09:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])
* 09:10 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 ([[phab:T376211|T376211]])

=== 2024-10-01 ===
* 15:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-30 ===
* 20:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T372814|T372814]])
* 16:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 16:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 13:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 13:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T372814|T372814]])
* 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 13:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 11:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 11:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 11:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 11:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
* 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
* 10:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
* 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 10:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=99)
* 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
* 09:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=99)
* 09:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
* 09:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 09:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-27 ===
* 15:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:56 arturo: [codfw1dev] restart rabbitmq again
* 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:15 arturo: [codfw1dev] restart rabbitmq
* 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:04 arturo: [codfw1dev] enable IPv6 on the neutron virtual router [[phab:T375847|T375847]]

=== 2024-09-26 ===
* 14:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 14:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T372814|T372814]])
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T372814|T372814]])
* 10:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T372814|T372814]])
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 08:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 08:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:47 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:45 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])
* 07:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T372814|T372814]])
* 07:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T372814|T372814]])

=== 2024-09-25 ===
* 14:11 arturo: [codfw1dev] start proxy-02 vm on proxy-codfw1dev project, it was in shutoff mode for unknown reasons
* 10:33 arturo: [codfw1dev] cleanup unused security groups ([[phab:T375604|T375604]])
* 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:48 arturo: [codfw1dev] restart rabbitmq on all cloudcontrol servers
* 09:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:35 arturo: [codfw1dev] deletre a bunch of tests and seemingly unused projects

=== 2024-09-24 ===
* 19:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 16:00 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 14:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T348643|T348643]])
* 14:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 14:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:56 arturo: [codfw1dev] restart rabbitmq-server on all 3 nodes
* 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack
* 09:10 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_rack (exit_code=97)
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack

=== 2024-09-23 ===
* 19:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_rack (exit_code=99)
* 15:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 15:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:42 arturo: put cloudvirt1048 in the network-ovs aggregate [[phab:T364457|T364457]]
* 14:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:37 arturo: [codfw1dev] restart rabbitmq
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:23 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

=== 2024-09-21 ===
* 10:17 dhinus: nova host-evacuate cloudvirt1063 ([[phab:T375223|T375223]])
* 09:59 dhinus: openstack aggregate remove host ceph cloudvirt1063 ([[phab:T375223|T375223]])
* 09:59 dhinus: openstack aggregate add host maintenance cloudvirt1063 ([[phab:T375223|T375223]])
* 09:42 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1063.eqiad.wmnet'
* 09:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1063.eqiad.wmnet'

=== 2024-09-20 ===
* 11:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 10:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 08:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T373740|T373740]])
* 08:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T373740|T373740]])

=== 2024-09-19 ===
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T373740|T373740]])
* 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T373740|T373740]])
* 15:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) ([[phab:T373740|T373740]])
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance ([[phab:T373740|T373740]])
* 10:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51
* 10:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51
* 09:56 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T374043|T374043]])
* 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 09:51 arturo: [codfw1dev] play with neutron default security group rules (delete, create them, etc) [[phab:T375111|T375111]]
* 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50

=== 2024-09-18 ===
* 15:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:21 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
* 15:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 15:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
* 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/47
* 11:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/47
* 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:40 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:39 dcaro: restarted rabbitmq-server on all cloudrabbits
* 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-09-17 ===
* 21:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T374043|T374043]])
* 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 16:11 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T374043|T374043]])
* 16:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T374043|T374043]])
* 15:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46

=== 2024-09-16 ===
* 14:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
* 11:28 arturo: [codfw1dev] created VM bastion-codfw1dev-04 to replace current bastion -03 ([[phab:T374828|T374828]])

=== 2024-09-12 ===
* 10:51 arturo: merging change to keystone wmf hooks https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071230 ([[phab:T374020|T374020]])

=== 2024-09-11 ===
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 16:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 16:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 16:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 15:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 15:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 15:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
* 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
* 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/41
* 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/41
* 11:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 07:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T374467|T374467]])
* 07:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T374467|T374467]])

=== 2024-09-10 ===
* 12:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 11:37 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 10:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:42 dcaro: hard-rebooting cloudvirt2004-dev (codfw1dev) having io/hardware issues
* 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 09:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
* 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 08:45 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-09 ===
* 21:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T373986|T373986]])
* 16:36 dcaro: cleaned up dns leaks
* 15:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/39
* 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/39
* 15:05 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 14:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:06 arturo: merged change to cloudgw NAT setting https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071189
* 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 12:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:42 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:38 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
* 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-06 ===
* 23:18 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T373986|T373986]])
* 18:17 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 13:46 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 07:27 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])

=== 2024-09-05 ===
* 21:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T374043|T374043]])
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 17:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T374043|T374043]])
* 17:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T374043|T374043]])
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T374043|T374043]])
* 14:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 14:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:19 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 14:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
* 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 12:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/36
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/36
* 10:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/35
* 10:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/35
* 09:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/34
* 09:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/34
* 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/33
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/33
* 09:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
* 09:22 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/31
* 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/31
* 09:16 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 08:46 arturo: [codfw1dev] restart rabbitmq @ codfw1dev [[phab:T374002|T374002]]

=== 2024-09-04 ===
* 21:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T373986|T373986]])
* 19:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:35 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T373986|T373986]])
* 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
* 12:18 arturo: [codfw1dev] restart rabbitmq-server.service on all 3 cloudcontrols, all nova-compute agents are down complaining about rabbitmq being unreachable

=== 2024-09-02 ===
* 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29
* 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29
* 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 11:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
* 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28

=== 2024-08-31 ===
* 13:55 andrewbogott: moving tools-redis-7 off of cloudvirt1048 just in case  [[phab:T373740|T373740]]
* 13:39 andrewbogott: rebooting cloudvirt1048 from mgmt, it seems to have crashed

=== 2024-08-30 ===
* 12:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-29 ===
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-08-25 ===
* 22:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-23 ===
* 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T369044|T369044]])
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T369044|T369044]])
* 00:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-22 ===
* 23:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=97) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T369044|T369044]])
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.unset_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.unset_maintenance ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=97) on host 'cloudvirt1039' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1037' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1035' ([[phab:T369044|T369044]])
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035' ([[phab:T369044|T369044]])
* 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1033' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1052' ([[phab:T369044|T369044]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052' ([[phab:T369044|T369044]])
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T369044|T369044]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T369044|T369044]])
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T369044|T369044]])
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])

=== 2024-08-20 ===
* 03:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)

=== 2024-08-19 ===
* 23:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 23:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 23:16 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-08-17 ===
* 03:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-16 ===
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:20 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
* 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
* 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T363344|T363344]])
* 05:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 04:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 04:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:57 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T363344|T363344]])
* 03:50 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 03:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T363344|T363344]])
* 01:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:12 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-15 ===
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:27 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 16:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 08:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 04:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 03:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 03:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 02:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 01:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2024-08-14 ===
* 23:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 23:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 23:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:05 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 04:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-08-12 ===
* 15:44 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) ([[phab:T363344|T363344]])
* 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T363344|T363344]])
* 08:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=0)
* 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:37 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97) ([[phab:T371878|T371878]])
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
* 08:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) ([[phab:T371878|T371878]])
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) ([[phab:T371878|T371878]])
* 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack ([[phab:T371878|T371878]])
* 08:17 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97)
* 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_rack

=== 2024-08-09 ===
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 18:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 13:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 13:38 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 13:35 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 13:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 11:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 05:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 05:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 00:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 00:47 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) ([[phab:T371878|T371878]])
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])

=== 2024-08-08 ===
* 23:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet'
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
* 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet'
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
* 18:39 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet'
* 18:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet'
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T371878|T371878]])
* 18:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet'
* 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet'
* 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet'
* 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet'
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet'
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
* 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
* 16:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet'
* 16:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet'
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudbackup1002-dev.codfw.wmnet'
* 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.codfw.wmnet'
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet'
* 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet'
* 16:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2006-dev.codfw.wmnet'
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2006-dev.codfw.wmnet'
* 16:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet'
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
* 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
* 15:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
* 15:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.wikimedia.org'
* 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.wikimedia.org'
* 15:51 andrewbogott: upgrading codfw1dev to openstack version caracal https://phabricator.wikimedia.org/T369044
* 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T369044|T369044]])
* 15:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 15:49 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T369044|T369044]])
* 15:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T369044|T369044]])
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 14:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 14:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.wait_for_rebalance
* 13:00 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
* 12:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 12:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 12:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 11:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T371878|T371878]])
* 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T371878|T371878]])
* 09:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 09:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 07:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 07:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
* 06:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 05:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 05:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 00:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 00:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)

=== 2024-08-07 ===
* 20:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 20:11 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=97)
* 20:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 20:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 20:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 18:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 17:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 17:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 17:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:56 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=97)
* 16:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node
* 16:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node
* 16:40 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:24 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 15:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T371878|T371878]])
* 08:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T371878|T371878]])
* 06:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 03:08 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])

=== 2024-08-06 ===
* 21:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 19:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 18:36 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet'
* 18:21 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1042.eqiad.wmnet'
* 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet'
* 18:17 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet'
* 18:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet'
* 17:59 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet'
* 17:58 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet'
* 17:45 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet'
* 17:43 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet'
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet'
* 17:32 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet'
* 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1045.eqiad.wmnet'
* 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet'
* 17:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet'
* 17:21 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet'
* 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet'
* 17:14 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet'
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet'
* 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 17:01 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet'
* 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 17:00 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:48 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:47 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 16:47 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:37 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 16:37 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 16:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 16:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T371878|T371878]])
* 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T371878|T371878]])
* 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:25 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
* 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance

=== 2024-08-01 ===
* 13:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-07-31 ===
* 23:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 23:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 16:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 15:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
* 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27

=== 2024-07-30 ===
* 11:29 arturo: installing nova security updates ([[phab:T371240|T371240]])

=== 2024-07-29 ===
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:28 arturo: [codfw1dev] restarting rabbitmq-server on all cloudcontrols, nova-compute cannot contact it
* 11:00 arturo: [codfw1dev] installing nova security updates ([[phab:T371240|T371240]])
* 10:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27

=== 2024-07-25 ===
* 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 14:46 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:45 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 14:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:09 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:55 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
* 11:40 arturo: manually restart maintain-dbusers in cloudcontrol1005 to see if that makes any difference
* 11:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 10:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
* 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 09:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 08:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 08:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
* 08:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 08:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 07:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 07:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23

=== 2024-07-24 ===
* 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 16:01 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 15:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:46 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:43 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:37 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 15:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
* 11:59 arturo: restarted maintain-dbusers.service in cloudcontrol1005, it was stuck doing nothing
* 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:43 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:42 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
* 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24

=== 2024-07-23 ===
* 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 10:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
* 10:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 10:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 10:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 10:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 09:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 05:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 05:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 05:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 05:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 03:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 03:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 02:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 02:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)

=== 2024-07-22 ===
* 23:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 23:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 21:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 20:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 18:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
* 15:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 15:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 15:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 14:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 13:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 13:28 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 13:28 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:10 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 12:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
* 11:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
* 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
* 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
* 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 11:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
* 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
* 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy

=== 2024-07-21 ===
* 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 09:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 06:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 06:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 03:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 03:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 00:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 00:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 00:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-20 ===
* 21:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 21:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 18:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 18:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 18:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 06:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 06:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 03:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 03:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 01:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 00:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 00:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-19 ===
* 22:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 21:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 21:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 19:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 18:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 18:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 06:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 06:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 01:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-18 ===
* 22:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 22:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 19:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 19:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 13:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 13:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 12:31 dhinus: upgrade spicerack from 8.5 to 8.8 on cloudcumin*
* 10:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 07:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 07:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 07:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 07:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 04:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 04:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 03:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 01:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-17 ===
* 22:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 22:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 17:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 07:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 05:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 05:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 04:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 04:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 02:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 02:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 01:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 01:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

=== 2024-07-16 ===
* 23:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 23:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 22:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 22:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 19:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 19:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 17:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 17:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 16:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 16:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 13:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 13:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 11:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 11:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 11:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 10:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
* 09:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 09:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 09:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 08:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 08:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 08:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 08:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-07-15 ===
* 22:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:40 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 20:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 20:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 20:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 18:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 18:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 17:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
* 16:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 14:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 14:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy

=== 2024-07-11 ===
* 13:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 13:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
* 12:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 11:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 11:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 02:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
* 01:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
* 01:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'

=== 2024-07-08 ===
* 17:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) ([[phab:T309789|T309789]])
* 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-07-06 ===
* 14:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-07-05 ===
* 10:33 arturo: aborrero@cloudcephmon1001:~$ sudo ceph osd unset norebalance
* 10:31 arturo: aborrero@cloudcephmon1001:~$ sudo ceph osd unset noin
* 08:56 arturo: installing nova/glance/cinder security updates [[phab:T369138|T369138]]

=== 2024-07-04 ===
* 20:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 20:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:52 dcaro: rebooting cloudcontrol1007 due to systemd-journal service failing to start
* 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-03 ===
* 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 16:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 12:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-02 ===
* 19:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])

=== 2024-07-01 ===
* 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 14:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 12:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-06-27 ===
* 22:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet'
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1059.eqiad.wmnet'
* 19:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1059.eqiad.wmnet'
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1059.eqiad.wmnet'
* 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet'
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1064.eqiad.wmnet'
* 17:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet'
* 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1067.eqiad.wmnet'
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet'
* 17:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1058.eqiad.wmnet'
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet'
* 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1066.eqiad.wmnet'
* 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet'
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1065.eqiad.wmnet'
* 17:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet'
* 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1057.eqiad.wmnet'
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) ([[phab:T309789|T309789]])
* 15:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 13:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 12:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet'
* 12:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirtlocal1001.eqiad.wmnet'
* 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 12:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 12:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 10:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 10:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 10:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 08:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 08:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 07:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
* 07:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2024-06-26 ===
* 22:32 andrewbogott: disabled all g3.* flavors in eqiad1
* 19:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 17:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 15:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) ([[phab:T309789|T309789]])
* 14:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add ([[phab:T309789|T309789]])
* 11:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) ([[phab:T309789|T309789]])
* 10:03 dcaro: taking cloudcephosd1006 out of the pool ([[phab:T348643|T348643]])
* 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])
* 10:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) ([[phab:T309789|T309789]])
* 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy ([[phab:T309789|T309789]])

=== 2024-06-25 ===
* 22:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2003-dev.codfw.wmnet'
* 22:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2003-dev.codfw.wmnet'
* 22:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2002-dev.codfw.wmnet'
* 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2002-dev.codfw.wmnet'
* 22:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codfw.wmnet'
* 22:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2001-dev.codw.wmnet'
* 22:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codw.wmnet'
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2005-dev.codfw.wmnet'
* 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2004-dev.codfw.wmnet'
* 16:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt2006-dev.codfw.wmnet'
* 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2006-dev.codfw.wmnet'
* 03:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 03:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-24 ===
* 19:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet'
* 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056.eqiad.wmnet'
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet'
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet'
* 16:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet'
* 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet'

=== 2024-06-21 ===
* 10:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:35 wmbot~arturo@nostromo: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97)
* 10:35 wmbot~arturo@nostromo: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 10:35 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:35 wmbot~arturo@nostromo: START - Cookbook wmcs.openstack.cloudvirt.vm_console
* 09:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 09:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 08:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T368129|T368129]])
* 08:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T368129|T368129]])
* 04:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 4e612eb8-04e1-4541-941d-{{Gerrit|a05519eed60a}}
* 04:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 4e612eb8-04e1-4541-941d-{{Gerrit|a05519eed60a}}

=== 2024-06-20 ===
* 21:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 14789ac1-bc06-4677-9bb0-{{Gerrit|66c16c887427}}
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 14789ac1-bc06-4677-9bb0-{{Gerrit|66c16c887427}}
* 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet'
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet'
* 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet'
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1052.eqiad.wmnet'
* 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet'
* 13:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 614f9c99-86f1-410f-8ef5-{{Gerrit|e33d23215ff5}}
* 13:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 614f9c99-86f1-410f-8ef5-{{Gerrit|e33d23215ff5}}
* 13:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1051.eqiad.wmnet'
* 13:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet'
* 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1050.eqiad.wmnet'
* 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet'
* 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049.eqiad.wmnet'
* 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet'
* 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet'
* 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet'
* 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet'
* 10:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet'
* 09:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet'
* 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet'
* 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1045.eqiad.wmnet'
* 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 04:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server d0b1d9d5-1aec-4a05-a2d7-{{Gerrit|6d8522a365dc}}
* 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server d0b1d9d5-1aec-4a05-a2d7-{{Gerrit|6d8522a365dc}}
* 04:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 2d2e8925-b50f-483a-82ee-{{Gerrit|e6a1c588e5be}}
* 04:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 2d2e8925-b50f-483a-82ee-{{Gerrit|e6a1c588e5be}}
* 04:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 138be95d-93ad-4a85-9245-{{Gerrit|a0a508711555}}
* 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 138be95d-93ad-4a85-9245-{{Gerrit|a0a508711555}}
* 04:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server a3e945dc-3548-47fa-8ce3-{{Gerrit|bf1426ff3b15}}
* 04:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 0258b810-29af-448a-af5e-{{Gerrit|ed39e19286df}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server a3e945dc-3548-47fa-8ce3-{{Gerrit|bf1426ff3b15}}
* 04:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 37e23659-4516-4bd8-a9be-{{Gerrit|4cc55def5560}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 0258b810-29af-448a-af5e-{{Gerrit|ed39e19286df}}
* 04:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 7da824f3-c9f3-4460-b9b2-{{Gerrit|2a894268a7ca}}
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 37e23659-4516-4bd8-a9be-{{Gerrit|4cc55def5560}}
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 63b82d38-3026-408f-8dcc-{{Gerrit|0ecbd1a3c870}}
* 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 7da824f3-c9f3-4460-b9b2-{{Gerrit|2a894268a7ca}}
* 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 7cb371bb-a53a-4e65-a1cf-{{Gerrit|f1a8264a9166}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 63b82d38-3026-408f-8dcc-{{Gerrit|0ecbd1a3c870}}
* 04:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 6616fcbf-a49e-4e03-b735-{{Gerrit|84d31b535c08}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 6616fcbf-a49e-4e03-b735-{{Gerrit|84d31b535c08}}
* 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 7cb371bb-a53a-4e65-a1cf-{{Gerrit|f1a8264a9166}}
* 04:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 6f1171db-9d7d-466f-aaa7-{{Gerrit|cb14e1a6af41}}
* 04:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 6f1171db-9d7d-466f-aaa7-{{Gerrit|cb14e1a6af41}}
* 04:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 1dc3fcee-cf27-4351-ad60-{{Gerrit|384478624ea3}}
* 04:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 77140f83-1a12-43b7-9e47-{{Gerrit|0e779503a525}}
* 04:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 1dc3fcee-cf27-4351-ad60-{{Gerrit|384478624ea3}}
* 04:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 3df62375-e75d-4068-9c71-{{Gerrit|13519b6bf927}}
* 04:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 3af9b40f-d29d-4216-9b64-{{Gerrit|0ebb10f94c7c}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 3df62375-e75d-4068-9c71-{{Gerrit|13519b6bf927}}
* 04:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 270b6533-dc99-4e5d-a642-{{Gerrit|c61138b11891}}
* 04:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server ae9fa949-e4ce-4ffb-ad9f-{{Gerrit|4e5a3812d031}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server ae9fa949-e4ce-4ffb-ad9f-{{Gerrit|4e5a3812d031}}
* 04:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 452dc8d3-6ee0-412c-90ba-{{Gerrit|66f21f9d09c1}}
* 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 3af9b40f-d29d-4216-9b64-{{Gerrit|0ebb10f94c7c}}
* 04:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 452dc8d3-6ee0-412c-90ba-{{Gerrit|66f21f9d09c1}}
* 04:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server e94378df-7a48-4b11-b44a-{{Gerrit|bf69aaf132bd}}
* 04:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 47b0da1d-e50a-42c1-8cd9-{{Gerrit|dc255cc2f1a3}}
* 04:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 47b0da1d-e50a-42c1-8cd9-{{Gerrit|dc255cc2f1a3}}
* 02:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T368007|T368007]])
* 02:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T368007|T368007]])
* 02:05 andrewbogott: cloudvirt1063 is unresponsive, cycling power from racadm

=== 2024-06-19 ===
* 15:43 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1044.eqiad.wmnet'
* 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 15:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet'
* 12:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
* 12:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet'
* 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet'
* 11:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet'
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1042.eqiad.wmnet'
* 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server f667d3c2-379a-48d7-ad44-{{Gerrit|4f3933bdb871}}
* 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server f667d3c2-379a-48d7-ad44-{{Gerrit|4f3933bdb871}}
* 01:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 71e4296d-039d-4452-9d92-{{Gerrit|69b9f8eb3aba}}
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 71e4296d-039d-4452-9d92-{{Gerrit|69b9f8eb3aba}}
* 01:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server a7725cd2-6162-41a3-8add-{{Gerrit|4dc0668b233b}}
* 01:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server a7725cd2-6162-41a3-8add-{{Gerrit|4dc0668b233b}}
* 01:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}
* 01:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 02bb9b5a-cadf-4bee-9b63-{{Gerrit|519b1e9b485b}}

=== 2024-06-18 ===
* 21:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudcontrol1006.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2024-06-17 ===
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T364457|T364457]])
* 20:03 andrewbogott: repaced ovs hosts in the 'ceph' aggregate
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T364457|T364457]])
* 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T364457|T364457]])
* 18:16 andrewbogott: temporarily removing all ovs hosts from the 'ceph' aggregate so the scheduler will stop putting linuxbridge hosts on ovs hosts and breaking them
* 17:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T364457|T364457]])
* 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T364457|T364457]])
* 13:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
* 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet'

=== 2024-06-14 ===
* 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet'
* 13:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet'

=== 2024-06-13 ===
* 16:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 16:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet'
* 13:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet'
* 13:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2003-dev.codfw.wmnet'
* 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2003-dev.codfw.wmnet'

=== 2024-06-12 ===
* 11:49 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1031.eqiad.wmnet'
* 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet'
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet'
* 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet'
* 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet'
* 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet'

=== 2024-06-11 ===
* 15:55 dcaro: restarting mon service on cloudcephmon1002 to try to release the 2 stuck ops left
* 13:30 taavi: pin all existing eqiad1 flavors to linuxbridge hypervisors [[phab:T364458|T364458]]
* 12:28 taavi: add all existing eqiad1 cloudvirts to new network-linuxbridge aggregate [[phab:T364458|T364458]]
* 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 10:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-07 ===
* 19:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-06-05 ===
* 11:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
* 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

=== 2024-06-04 ===
* 15:49 taavi: drop hopefully-unused 68.10.in-addr.arpa. from designate [[phab:T361220|T361220]]

=== 2024-05-30 ===
* 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-29 ===
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T364984|T364984]])
* 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T364984|T364984]])
* 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T364984|T364984]])
* 18:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T364984|T364984]])

=== 2024-05-28 ===
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 14:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-21 ===
* 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 08:18 taavi: stop neutron services on cloudnet1005 [[phab:T364459|T364459]]

=== 2024-05-20 ===
* 14:46 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:46 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:20 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
* 14:19 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:18 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:18 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:17 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:17 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:16 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:16 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
* 14:15 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
* 14:15 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs

=== 2024-05-16 ===
* 08:55 taavi: delete 'monitoring' project https://phabricator.wikimedia.org/T365105

=== 2024-05-15 ===
* 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T319184|T319184]])
* 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-05-14 ===
* 19:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-05-10 ===
* 13:23 andrewbogott: deploying updated 2024.1 Horizon

=== 2024-05-09 ===
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-25 ===
* 21:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T356287|T356287]])
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' ([[phab:T356287|T356287]])
* 21:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T356287|T356287]])
* 20:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T356287|T356287]])
* 18:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1033' ([[phab:T356287|T356287]])
* 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033' ([[phab:T356287|T356287]])
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 17:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T356287|T356287]])
* 16:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T356287|T356287]])
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T356287|T356287]])

=== 2024-04-23 ===
* 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2002-dev.codfw.wmnet'
* 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2002-dev.codfw.wmnet'

=== 2024-04-18 ===
* 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2008-dev.codfw.wmnet'
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2008-dev.codfw.wmnet'

=== 2024-04-17 ===
* 14:12 dcaro: deleting dns leaks
* 01:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 01:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])

=== 2024-04-16 ===
* 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 20:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2001-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2001-dev.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' ([[phab:T356287|T356287]][A)
* 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.codfw.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.eqiad.wmnet' ([[phab:T356287|T356287]])
* 19:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) ([[phab:T356287|T356287]])
* 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T356287|T356287]])

=== 2024-04-15 ===
* 11:19 taavi: update spicerack to 8.5.0 on cloudcumin2001

=== 2024-04-10 ===
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-09 ===
* 18:13 andrewbogott: rebooting cloudinfra-cloudvps-puppetserver-1; unresponsive
* 13:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-05 ===
* 14:37 taavi: run maintain-replica-indexes on all web replicas [[phab:T361945|T361945]]
* 14:27 taavi: run maintain-replica-indexes on remaining analytics replicas [[phab:T361945|T361945]]
* 14:17 taavi: run maintain-replica-indexes on clouddb1017 [[phab:T361945|T361945]]

=== 2024-04-04 ===
* 18:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-04-03 ===
* 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 15:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T319184|T319184]])
* 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T319184|T319184]])
* 10:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' ([[phab:T319184|T319184]])
* 10:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T319184|T319184]])
* 09:17 taavi: manually delete prometheus-node-textfile-wmcs-dnsleaks.service and related files from cloudservices1005/6, leftovers of the designate api to cloudcontrol migration
* 09:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' ([[phab:T319184|T319184]])
* 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2024-04-02 ===
* 15:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T319184|T319184]])
* 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet' ([[phab:T319184|T319184]])
* 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 12:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 12:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
* 12:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-03-27 ===
* 11:47 taavi: deleting about 2k stale puppet certs by running wmcs-puppetcertleaks in delete mode

=== 2024-03-22 ===
* 10:25 dcaro: adding back cloudcephosd1034 to the pool after doing the performance tests ([[phab:T348643|T348643]])

=== 2024-03-21 ===
* 22:07 andrewbogott: doing dist-upgrade on cloudcontrol nodes to get mariadb upgraded for [[phab:T357133|T357133]]
* 22:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T357133|T357133]])
* 22:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' ([[phab:T357133|T357133]])
* 11:08 dcaro: restarting nova-api on cloudcontrol1007

=== 2024-03-20 ===
* 17:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:27 dcaro: turning off cloudcephosd1030 to swap some disks ([[phab:T348643|T348643]])
* 15:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T348643|T348643]])
* 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2024-03-19 ===
* 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 15:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])
* 15:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) ([[phab:T348643|T348643]])
* 13:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T348643|T348643]])

=== 2024-03-18 ===
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-03-09 ===
* 16:43 andrewbogott: restarted nova-api on cloudcontrol1006

=== 2024-03-08 ===
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:21 arturo: restarted nova-api in cloudcontrol1007, it was complaining about mysql broken pipe
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 10:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 10:35 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 10:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

=== 2024-03-07 ===
* 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codfw.wmnet'
* 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
* 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance

=== 2024-03-06 ===
* 17:46 dhinus: running &quot;wmcs-dnsleaks --delete&quot; to clean up 2 leaked records (tools-sgeweblight-10-32)
* 15:36 dcaro: renewing puppet ca cert for cloud-puppetmaster-03
* 15:24 dcaro: renewing puppet ca cert for cloudinfra-internal puppetmaster

=== 2024-03-04 ===
* 14:56 dhinus: delete project &quot;loggerdiscordbot&quot; in favor of new project &quot;discordbots&quot; [[phab:T358337|T358337]],[[phab:T358427|T358427]]
* 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reboot_node (exit_code=0) ([[phab:T359049|T359049]])
* 12:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reboot_node ([[phab:T359049|T359049]])

=== 2024-03-01 ===
* 14:59 taavi: removing wmf-auto-restart-cron from all VMs without cron via cumin - https://gerrit.wikimedia.org/r/c/operations/puppet/+/1007328/ [[phab:T358343|T358343]]
* 12:07 dcaro: restarted nova-api on cloudcontrol100* as it was very slow
* 12:04 dcaro: restarted nova-api on cloudcontrol1005 as it was very slow

=== 2024-02-27 ===
* 18:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-26 ===
* 10:02 arturo: deleting nskaggs account from gerrit's wmcs-trusted group

=== 2024-02-22 ===
* 13:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 09:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-21 ===
* 13:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) ([[phab:T319184|T319184]])
* 13:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance ([[phab:T319184|T319184]])
* 13:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T319184|T319184]])
* 13:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T319184|T319184]])
* 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 12:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' ([[phab:T319184|T319184]])

=== 2024-02-20 ===
* 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:45 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=0) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])

=== 2024-02-19 ===
* 12:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. ([[phab:T319184|T319184]])
* 10:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=0) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=99) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])
* 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) ([[phab:T319184|T319184]])

=== 2024-02-15 ===
* 17:34 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.roll_reboot_cloudnets ([[phab:T356975|T356975]])
* 15:34 taavi: restart radosgw in eqiad as I am seeing 500 errors
* 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T319184|T319184]])
* 11:58 dhinus: restore correct aggregate &quot;localdisk&quot; for cloudvirtlocal1001 and remove &quot;maintenance&quot;
* 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T319184|T319184]])
* 05:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1045.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1044.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 05:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1047.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet[B&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1046.eqiad.wmnet[B&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1049.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1048.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1051.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1050.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 04:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1052.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1053.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1054.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1055.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'

=== 2024-02-14 ===
* 22:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1057.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1056.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1058.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1059.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1061.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1060.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1062.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1064.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1065.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1066.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1067.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtXXXX.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtXXXX.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:49 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:49 wmbot~andrew@bullseye: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97)
* 19:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:48 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:45 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:45 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1063.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 19:38 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:38 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:36 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:36 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:34 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:34 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 19:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 19:32 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 17:13 wmbot~fran@wmf3169: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtlocal1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T356975|T356975]])
* 17:12 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirtlocal1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;' ([[phab:T356975|T356975]])
* 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1043.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1042.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1041.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1040.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:09 taavi: creating some missing $PROJECT.wmcloud.org. DNS zones
* 14:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1039.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 14:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1038.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1037.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1036.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 13:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1035.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1034.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 12:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1033.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1032.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 11:21 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 10:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 09:31 taavi: failover all dumps traffic to clouddumps1001
* 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::codfw1dev::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 08:16 taavi: reboot clouddumps1001 for kernel updates

=== 2024-02-13 ===
* 17:07 wmbot~taavi@runko: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 17:07 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:06 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:06 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:04 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;O:wmcs::openstack::eqiad1::virt_ceph&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::eqiad1::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P&lt;nowiki&gt;{&lt;/nowiki&gt;P:openstack::eqiad1::nova::compute::service&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:02 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P:openstack::eqiad1::nova::compute::service'
* 16:02 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P:openstack::eqiad1::nova::compute::service'
* 16:01 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 16:01 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D&lt;nowiki&gt;{&lt;/nowiki&gt;cloudvirt1001.eqiad.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt;'
* 15:10 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) ([[phab:T356975|T356975]])
* 14:55 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T356975|T356975]])

=== 2024-02-12 ===
* 17:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0)
* 17:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0)
* 17:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:23 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99)
* 17:21 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 17:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 17:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 17:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:36 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 16:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 16:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 15:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
* 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 15:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99)
* 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
* 01:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:34 andrewbogott: resetting eqiad1 rabbitmq in hopes of resolving neutron double message warnings
* 01:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 00:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 00:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-11 ===
* 21:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:57 andrewbogott: running wmcs.openstack.restart_openstack for all eqiad1 services
* 20:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 20:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-08 ===
* 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:43 taavi: deploy change to exclude cloud-private networks from general egress NAT https://phabricator.wikimedia.org/T356850

=== 2024-02-06 ===
* 20:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:09 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 02:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-02-05 ===
* 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:55 andrewbogott: rebuilt bookworm base image in eqiad1 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/992677

=== 2024-02-02 ===
* 13:54 arturo: [codfw1dev] cleanup /etc/network/interfaces on cloudlb2003-dev from puppet leftovers

=== 2024-02-01 ===
* 10:55 taavi: invite aborrero to /repos/cloud, /toolforge-repos, /cloudvps-repos on gitlab

=== 2024-01-29 ===
* 12:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-26 ===
* 09:01 taavi: joining cloudrabbit1001/2 to the cluster on 1003 [[phab:T345610|T345610]]

=== 2024-01-25 ===
* 16:48 andrewbogott: taavi just moved all rabbitmq traffic to cloudrabbit1003 as part of [[phab:T345610|T345610]]
* 16:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:27 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:27 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-2.tools.eqiad1.wikimedia.cloud to the cluster
* 13:15 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
* 13:15 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-1.tools.eqiad1.wikimedia.cloud to the cluster
* 12:48 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
* 12:48 wmbot~taavi@runko: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-1.toolsbeta.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-24 ===
* 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
* 11:37 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-22 ===
* 16:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:56 andrewbogott: restarting openstack sevices on eqiad1 to clean up from the mariadb restarts
* 15:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:07 andrewbogott: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/992192 and resetting galera cluster in eqiad1

=== 2024-01-21 ===
* 05:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 05:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-19 ===
* 23:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-18 ===
* 20:10 taavi: mysql:labsdbaccounts@m5-master.eqiad.wmnet [labsdbaccounts]&gt; update account_host set status = 'absent' where id = 137613;
* 12:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 12:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-101.tools.eqiad1.wikimedia.cloud to the cluster

=== 2024-01-17 ===
* 15:17 andrewbogott: &quot;systemctl restart mariadb@s4.service mariadb@s6.service&quot; on clouddb1015. System is in danger of oom and there are no obvious long queries running

=== 2024-01-16 ===
* 14:25 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 14:25 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 14:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 13:55 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:55 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:54 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:54 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:53 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
* 13:53 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 13:52 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
* 13:52 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
* 11:35 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
* 11:34 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
* 11:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 09:41 taavi: drop dbproxy1018/9 grants from all clouddb hosts [[phab:T346947|T346947]]
* 09:24 taavi: move cloudvirt2004-dev from 'failed' to 'active' in netbox - seems like that was for [[phab:T348531|T348531]] which is now resolved

=== 2024-01-15 ===
* 14:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 14:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 14:32 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) ([[phab:T355061|T355061]])
* 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot ([[phab:T355061|T355061]])
* 12:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2024-01-11 ===
* 10:51 dcaro: restarting striker.service on cloudweb1003 as it seems non-responsive

=== 2024-01-10 ===
* 17:43 bd808: Blocking Developer accounts connected to invalid/legacy wikimedia.org email addresses ([[phab:T218239|T218239]])
* 16:48 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0) ([[phab:T346631|T346631]])
* 16:48 wmbot~fran@wmf3169: test message3 from local cookbook ([[phab:T346631|T346631]])
* 16:48 wmbot~fran@wmf3169: START - Cookbook wmcs.do_log_msg ([[phab:T346631|T346631]])

=== 2024-01-09 ===
* 17:50 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0) ([[phab:T346631|T346631]])
* 17:49 wmbot~fran@wmf3169: test message2 from local cookbook ([[phab:T346631|T346631]])
* 17:49 wmbot~fran@wmf3169: START - Cookbook wmcs.do_log_msg ([[phab:T346631|T346631]])
* 17:43 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])
* 17:43 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])
* 17:42 wmbot~fran@wmf3169: %(message)s ([[phab:T346631|T346631]])

=== 2024-01-08 ===
* 15:52 taavi: verify wmcloud.org, wmflabs.org and toolforge.org in gmail postmaster console to figure out how much google likes us ([[phab:T354112|T354112]])

=== 2024-01-07 ===
* 19:34 andrewbogott: removed cloudvirt1063 from 'ceph' aggregate, added to 'maintenance' aggregate   [[phab:T353408|T353408]]
* 19:34 andrewbogott: evacuating all VMs from cloudvirt1063.  [[phab:T353408|T353408]]

=== 2024-01-02 ===
* 16:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T353408|T353408]])
* 16:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T353408|T353408]])
* 10:22 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
* 10:22 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.vm_console

=== 2023-12-31 ===
* 21:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:35 andrewbogott: running openstack service restart cookbook in eqiad1 in response to a bunch of service down alerts
* 21:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-21 ===
* 16:52 dhinus: puppet node deactivate cloudvirt1063.eqiad.wmnet [[phab:T353406|T353406]]
* 03:01 andrewbogott: restarting mariadb on cloudcontrol1005, hoping to get Galera back in sync

=== 2023-12-20 ===
* 19:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-18 ===
* 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:23 andrewbogott: restarting all eqiad1 openstack services after a rabbitmq upgrade/rebuild for [[phab:T353646|T353646]]
* 15:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-15 ===
* 13:13 dcaro: restarted nova-fullstack on codfw as it was stuck (and alerting through stale prometheus file)

=== 2023-12-14 ===
* 00:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
* 00:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
* 00:25 andrewbogott: evacuating hosts from cloudvirt1063 and depooling. [[phab:T353406|T353406]]

=== 2023-12-13 ===
* 16:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=99)

=== 2023-12-12 ===
* 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 17:45 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-100.tools.eqiad1.wikimedia.cloud to the cluster
* 16:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 16:11 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-99.tools.eqiad1.wikimedia.cloud to the cluster
* 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 15:49 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-98.tools.eqiad1.wikimedia.cloud to the cluster

=== 2023-12-10 ===
* 18:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:27 andrewbogott: restarting all openstack API servers, hoping to make things a bit more responsive
* 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-08 ===
* 12:00 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99)  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud ([[phab:T353055|T353055]])
* 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud ([[phab:T353055|T353055]])
* 11:58 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=97)  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud
* 11:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs  on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud
* 09:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) ([[phab:T345084|T345084]])
* 09:32 dcaro: restarting nova and keystone as they are getting too slow ([[phab:T345084|T345084]])
* 09:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack ([[phab:T345084|T345084]])
* 09:32 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) ([[phab:T345084|T345084]])
* 09:31 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack ([[phab:T345084|T345084]])

=== 2023-12-07 ===
* 13:12 dcaro: rebooting cloudcephosd1001 to make sure puppet7 migration went ok

=== 2023-12-04 ===
* 00:08 andrewbogott: rebooting cloudcontrol1006 to recover from full disk error

=== 2023-12-03 ===
* 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-12-02 ===
* 12:27 taavi: powercycle cloudvirt1063 [[phab:T352595|T352595]]
* 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:28 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-97.tools.eqiad1.wikimedia.cloud to the cluster
* 11:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 11:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-96.tools.eqiad1.wikimedia.cloud to the cluster
* 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 10:50 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-95.tools.eqiad1.wikimedia.cloud to the cluster
* 00:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:21 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-94.tools.eqiad1.wikimedia.cloud to the cluster
* 00:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-93.tools.eqiad1.wikimedia.cloud to the cluster
* 00:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-92.tools.eqiad1.wikimedia.cloud to the cluster
* 00:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-91.tools.eqiad1.wikimedia.cloud to the cluster
* 00:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:05 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-90.tools.eqiad1.wikimedia.cloud to the cluster
* 00:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
* 00:01 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-89.tools.eqiad1.wikimedia.cloud to the cluster

=== 2023-12-01 ===
* 17:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:19 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T351171|T351171]])
* 16:19 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 15:49 andrewbogott: reimaging cloudcontrol1005 due to widespread misbehavior
* 14:24 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T351171|T351171]])
* 14:20 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 14:19 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97) ([[phab:T351171|T351171]])
* 14:18 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T351171|T351171]])
* 13:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:55 taavi: restart neutron-rpc-server.service on eqiad1 cloudcontrols

=== 2023-11-30 ===
* 20:44 andrewbogott: generating to application credentials for the tests that run on tf-infra-test
* 19:54 andrewbogott: reimaged cloudrabbit100[23] after https://gerrit.wikimedia.org/r/c/operations/puppet/+/979127.  I didn't reimage 1001 because that will require rebuilding the whole cluster but I did remove the related packages.
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet'
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet'
* 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
* 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet'
* 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet'
* 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet'
* 17:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet'
* 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet'
* 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet'
* 17:47 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet'
* 17:42 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:30 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:19 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:19 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:09 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' ([[phab:T348843|T348843]])
* 17:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:59 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:54 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:39 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:34 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:15 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:10 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:05 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T348843|T348843]])
* 16:00 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:31 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:22 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:17 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:17 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:13 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:07 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:38 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:16 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:03 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 13:55 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 13:43 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) ([[phab:T348843|T348843]])
* 13:43 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudweb.set_maintenance ([[phab:T348843|T348843]])
* 12:18 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 12:04 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:59 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:45 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:44 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T348843|T348843]])
* 11:28 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' ([[phab:T348843|T348843]])

=== 2023-11-29 ===
* 15:27 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 15:15 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' ([[phab:T348843|T348843]])
* 14:59 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T348843|T348843]])
* 14:50 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T348843|T348843]])

=== 2023-11-28 ===
* 14:18 taavi: moving wiki replica DNS to use cloudlbs instead of the old proxy VMs [[phab:T346947|T346947]]

=== 2023-11-27 ===
* 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-24 ===
* 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-22 ===
* 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 13:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-21 ===
* 10:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-20 ===
* 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-17 ===
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-16 ===
* 12:09 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 12:05 taavi@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
* 11:23 dhinus: upgraded spicerack from 8.0.2 to 8.0.3 on cloudcumins
* 05:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 05:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-15 ===
* 09:50 taavi: move cloudlb hosts to use the nftables firewall backend

=== 2023-11-14 ===
* 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T345811|T345811]])
* 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1052.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1050.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 18:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' ([[phab:T345811|T345811]])
* 12:03 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T345811|T345811]])
* 11:44 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet' ([[phab:T345811|T345811]])
* 10:10 taavi: restart kiwix-mirror-update on clouddumps1001
* 05:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 05:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 04:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 04:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 04:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 04:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 03:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 03:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 03:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 03:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 02:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])

=== 2023-11-13 ===
* 22:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 22:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 22:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 22:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 21:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 19:31 andrewbogott: rebooting cloudcontrol2005-dev, trying to fix general misbehavior
* 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) ([[phab:T345811|T345811]])
* 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 19:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])
* 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 18:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1033.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1032.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet'
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet'
* 17:09 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) ([[phab:T345811|T345811]])
* 17:09 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T345811|T345811]])
* 16:57 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance ([[phab:T345811|T345811]])
* 15:37 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T345811|T345811]])
* 15:19 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' ([[phab:T345811|T345811]])
* 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 09:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-11-11 ===
* 02:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 02:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 02:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 01:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 01:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
* 00:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 00:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 00:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1058'
* 00:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1058'
* 00:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 00:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot

=== 2023-11-09 ===
* 21:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:54 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1025.eqiad.wmnet' ([[phab:T345811|T345811]])
* 14:52 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1025.eqiad.wmnet' ([[phab:T345811|T345811]])
* 14:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345811|T345811]])
* 14:50 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345811|T345811]])

=== 2023-11-08 ===
* 16:43 andrewbogott: created foundationmemory project for [[phab:T350760|T350760]]

=== 2023-11-06 ===
* 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 18:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 15:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)

=== 2023-11-05 ===
* 00:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-11-04 ===
* 23:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 22:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 04:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)

=== 2023-11-03 ===
* 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:13 dhinus: triggering neutron failover from cloudnet1005 to cloudnet1006 ([[phab:T345811|T345811]])
* 06:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 01:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 01:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)

=== 2023-11-02 ===
* 20:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:32 taavi: merged cloudcontrol firewall cleanup patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/971211
* 16:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 16:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 13:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 13:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 11:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 11:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 07:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 07:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 05:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 03:47 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 03:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)

=== 2023-11-01 ===
* 23:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 21:29 taavi: re-enable puppet on cloudcontrol2006-dev.codfw.wmnet which has fallen off of puppetdb
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 20:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 19:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 14:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 14:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 09:04 taavi: reset local cookbook changes on cloudcumin1001 which were causing issues with puppet runs
* 09:02 taavi: restart nova-fullstack which had had some issues after yesterday's cloudcontrol1007 reimage
* 02:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 00:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-10-31 ===
* 23:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 18:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 14:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 14:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 14:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 10:16 dhinus: upgrading mariadb-server in cloudcontrol1005 ([[phab:T345811|T345811]])
* 10:04 dhinus: upgrading mariadb-server in cloudcontrol1006 ([[phab:T345811|T345811]])
* 09:51 dhinus: upgrading mariadb-server in cloudcontrol1007, second attempt ([[phab:T345811|T345811]])
* 06:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 05:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 02:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
* 02:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
* 02:27 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
* 02:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
* 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

=== 2023-10-30 ===
* 17:40 andrewbogott: rebooting tools-db-1.tools.eqiad1.wikimedia.cloud for yet another oom death
* 17:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 16:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
* 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:39 dhinus: upgrading mariadb-server in cloudcontrol1007 ([[phab:T345811|T345811]])
* 16:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) ([[phab:T348643|T348643]])
* 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node ([[phab:T348643|T348643]])
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
* 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-28 ===
* 07:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)

=== 2023-10-27 ===
* 15:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 15:38 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 15:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 15:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 15:29 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 15:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 15:27 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 15:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:09 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 09:05 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 09:04 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-26 ===
* 14:02 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 09:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-25 ===
* 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
* 11:00 taavi: update cloudcumins to spicerack 8.x
* 10:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster

=== 2023-10-24 ===
* 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-10-23 ===
* 16:48 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:06 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project catalyst in eqiad1
* 15:06 wm-bot2: fran@wmf3169 START - Cookbook wmcs.vps.create_project for project catalyst in eqiad1
* 10:36 taavi: merged change https://gerrit.wikimedia.org/r/c/operations/puppet/+/966494 which touches the pdns web server config

=== 2023-10-20 ===
* 15:17 dcaro: upgraded cloudcephosd1004 to v15 ([[phab:T349363|T349363]])
* 14:25 dcaro: upgraded cloudcephosd1003 to v15 ([[phab:T349363|T349363]])
* 13:20 dcaro: upgraded cloudcephosd1002 to v15 ([[phab:T349363|T349363]])
* 10:33 dcaro: upgraded cloudcephosd1001 to v15 ([[phab:T349363|T349363]])
* 08:26 dcaro: codfw ceph enabled diskprediction_local module, will take a bit to populate/start getting predictions ([[phab:T348716|T348716]])

=== 2023-10-17 ===
* 15:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T349109|T349109]])
* 15:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T349109|T349109]])

=== 2023-10-16 ===
* 03:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 03:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-10-13 ===
* 23:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 23:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 19:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 08:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 08:31 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 08:30 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 08:20 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])

=== 2023-10-12 ===
* 17:16 wm-bot2: fran@wmf3169 END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) ([[phab:T341285|T341285]])
* 17:16 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])

=== 2023-10-11 ===
* 10:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 10:36 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 07:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 06:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-10 ===
* 17:25 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 14:50 dcaro: removing ~100 dangling backup snapshots from eqiad1-compute ceph pool
* 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 12:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 11:38 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 11:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 11:33 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
* 11:32 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
* 11:00 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) ([[phab:T341285|T341285]])
* 11:00 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 10:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 10:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 09:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 09:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) ([[phab:T341285|T341285]])
* 09:43 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack ([[phab:T341285|T341285]])
* 08:19 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:18 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:18 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:17 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:17 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 08:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 08:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node

=== 2023-10-09 ===
* 17:16 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 16:26 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:18 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:49 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:26 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 13:55 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 13:45 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 13:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 13:04 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:10 dcaro: undrained cephosd1011
* 09:09 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 09:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:06 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 09:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:05 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 09:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 07:35 taavi: restart postgresql on cloudbackup2001 [[phab:T348431|T348431]]

=== 2023-10-05 ===
* 16:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:29 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 16:24 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 16:11 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 15:30 arturo: operating on cloudgw @ eqiad1 ([[phab:T347469|T347469]])
* 14:07 wm-bot2: dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
* 12:55 arturo: doing cloudgw maintenance operations [[phab:T347469|T347469]]
* 11:54 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_rack
* 10:57 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 09:54 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 09:52 arturo: [codfw1dev] aborrero@cloudcontrol2001-dev:~ $ sudo keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
* 09:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 09:40 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 09:33 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 08:50 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 00:04 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)

=== 2023-10-04 ===
* 20:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 15:55 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 14:54 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) ([[phab:T341285|T341285]])
* 14:44 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 14:41 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 14:40 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node ([[phab:T341285|T341285]])
* 13:50 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) ([[phab:T341285|T341285]])
* 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:08 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:21 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 07:12 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 07:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

=== 2023-10-03 ===
* 19:38 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:35 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:32 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 12:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 11:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 08:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:43 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
* 08:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:39 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 08:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 08:23 dcaro: set .rgw.root pool on eqiad as rgw app (`ceph osd pool application enable .rgw.root rgw`)

=== 2023-10-02 ===
* 17:39 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 16:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 14:30 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:58 taavi: cloudcontrol1005,7: `sudo systemctl reset-failed keystone_sync_keys_from_cloudcontrol1006.eqiad.wmnet.service`
* 13:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:40 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
* 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
* 13:37 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 12:26 arturo: [codfw1dev] run `update domains set master = '185.15.57.25:5354 185.15.57.26:5354 172.20.5.9:5354 172.20.5.8:5354';` in cloudservies2005-dev
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])
* 11:55 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])
* 11:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) ([[phab:T316544|T316544]])
* 08:44 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node ([[phab:T316544|T316544]])

=== 2023-09-29 ===
* 12:43 taavi: taavi@cloudcontrol1005 ~ $ os subnet set a69bdfad-d7d2-4cfa-8231-{{Gerrit|3d6d3e0074c9}} --no-dns-nameservers --dns-nameserver 172.20.255.1
* 08:36 taavi: start script to fix networking on broken bullseye instances [[phab:T347665|T347665]]

=== 2023-09-28 ===
* 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 18:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 12:01 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
* 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
* 09:48 arturo: rebooting cloudgw1001/1002 for sysctl and kernel upgrades

=== 2023-09-27 ===
* 12:07 arturo: merging cloudgw firewall changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/961360
* 09:36 taavi: move maintain-dbusers to cloudcontrol1005
* 01:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-26 ===
* 17:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-24 ===
* 15:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-23 ===
* 08:40 taavi: restart keystone

=== 2023-09-22 ===
* 14:03 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 14:02 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 14:02 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 14:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 14:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.undrain_node (exit_code=0)
* 14:00 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 14:00 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:56 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.undrain_node (exit_code=0)
* 13:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.undrain_node (exit_code=99)
* 13:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:48 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.undrain_node (exit_code=99)
* 13:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
* 13:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 13:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:46 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:44 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:41 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 13:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
* 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:37 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:33 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 12:33 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 12:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
* 11:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
* 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node

=== 2023-09-21 ===
* 16:35 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0)  on tools-db-3.tools.eqiad1.wikimedia.cloud
* 16:34 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs  on tools-db-3.tools.eqiad1.wikimedia.cloud
* 02:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
* 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-20 ===
* 21:49 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:45 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:38 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 21:35 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 21:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 21:31 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 11:12 arturo: moving openstack API endpoint to cloudlb ([[phab:T346439|T346439]])
* 10:28 arturo: running SQL command `update domains set master=&quot;172.20.1.5:5354 172.20.2.4:5354 185.15.56.162:5354 185.15.56.163:5354&quot;;` on cloudservices1005/1006 ([[phab:T346042|T346042]])
* 10:06 arturo: running SQL command `update domains set master=&quot;185.15.56.162:5354 185.15.56.163:5354&quot;` on cloudservices1005/1005 ([[phab:T346042|T346042]])

=== 2023-09-19 ===
* 18:54 andrewbogott: depooling clouddb1019 to let it recover from high memory use.  [[phab:T346826|T346826]]
* 15:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-18 ===
* 11:53 taavi: update designate urls in Keystone to point to openstack-next, until cloudlb is serving the main openstack address [[phab:T346042|T346042]]
* 11:40 arturo: decomission cloudservices1005 [[phab:T346042|T346042]] in preparation for re-racking
* 08:45 arturo: hardcode `185.15.56.161 openstack.eqiad1.wikimediacloud.org` in /etc/hosts in cloudcontrol1005 for [[phab:T346441|T346441]]

=== 2023-09-15 ===
* 11:43 arturo: merging NAT change for [[phab:T346426|T346426]] in cloudgw
* 10:33 arturo: faiolver cloudgw1001 into cloudgw1002, investigating a nftables syntax error ([[phab:T346432|T346432]])

=== 2023-09-14 ===
* 21:25 wm-bot2: andrew@bullseye END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:25 wm-bot2: andrew@bullseye START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:23 wm-bot2: andrew@bullseye END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:23 wm-bot2: andrew@bullseye START - Cookbook wmcs.openstack.cloudvirt.drain
* 21:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
* 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
* 17:13 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 17:06 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 17:01 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:51 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:29 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) ([[phab:T345810|T345810]])
* 16:29 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:25 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) ([[phab:T345810|T345810]])
* 16:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain ([[phab:T345810|T345810]])
* 16:12 topranks: DNS operation: remove old DNS entry for ns0.openstack.eqiad1.wikimediacloud.org. on wikimedia authdns (was pointing to 208.80.154.148)
* 16:10 topranks: DNS operation: add new DNS entry for ns0.openstack.eqiad1.wikimediacloud.org. on wikimedia authdns pointing to 185.15.56.162
* 14:42 arturo: DNS operation: route 208.80.154.148 to cloudservices1006 in anticipation of cloudservices1005 decom ([[phab:T346042|T346042]])
* 12:11 arturo: enable puppet on cloudservices1006 to drop local NAT hacks and enable new DNS auth IP address ([[phab:T346042|T346042]])

=== 2023-09-13 ===
* 17:11 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) ([[phab:T345811|T345811]])
* 17:08 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 17:04 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 17:04 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:57 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:53 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:53 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:49 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:49 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 16:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) ([[phab:T345811|T345811]])
* 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node ([[phab:T345811|T345811]])
* 01:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 01:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 01:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 01:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 01:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-12 ===
* 18:06 andrewbogott: update domains set master='185.15.56.163:5354 208.80.154.11:5354 10.64.151.4:5354'; on cloudservices1005 + cloudservices1006
* 17:59 andrewbogott: mysql:root@localhost [pdns]&gt; update domains set master='185.15.56.163:5354 208.80.154.11:5354';
* 17:59 andrewbogott: &quot;designate-manage pool update' on cloudservices1005 to remove cloudservices1004 from the pool
* 17:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-11 ===
* 12:36 arturo: update DNS resolver cloud-wide to use 172.20.255.1 ([[phab:T342621|T342621]])

=== 2023-09-10 ===
* 02:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 02:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 02:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
* 02:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-09-07 ===
* 10:09 dhinus: reimaging cloudcontrol2001-dev to bookworm ([[phab:T345810|T345810]])

=== 2023-09-06 ===
* 16:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97)
* 16:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:53 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:37 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:37 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:37 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:35 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:35 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:34 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
* 16:34 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:23 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97)
* 16:23 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
* 16:13 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99)
* 16:12 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance

=== 2023-09-05 ===
* 12:34 arturo: synced pdns database from cloudservices1004 to cloudservices1006 ([[phab:T345240|T345240]])
* 12:18 arturo: updating pools.yaml in all cloudservices designate nodes ([[phab:T345240|T345240]])
* 10:54 arturo: running SQL command `update domains set master=&quot;208.80.154.11:5354 208.80.154.148:5354 10.64.151.4:5354&quot;;` on all 3 cloudservices nodes ([[phab:T345240|T345240]])

=== 2023-09-04 ===
* 15:58 arturo: stop and mask designate-sink.service @ cloudservices1006
* 14:19 arturo: started all designate services on cloudservices1006 [[phab:T345240|T345240]]
* 10:46 arturo: added designate galera DB grants for cloudlb [[phab:T345240|T345240]]
* 08:40 arturo: stopped all designate services on cloudservices1006 [[phab:T345240|T345240]]

=== 2023-09-01 ===
* 10:28 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.network.tests (exit_code=1) ([[phab:T345282|T345282]])
* 10:25 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.network.tests ([[phab:T345282|T345282]])

=== 2023-08-31 ===
* 15:14 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=99)
* 15:14 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
* 12:49 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudnet.show (exit_code=0)
* 12:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudnet.show
* 12:48 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
* 12:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.unset_cluster_maintenance
* 12:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.set_cluster_in_maintenance
* 12:46 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
* 12:46 wm-bot2: fran@wmf3169 START - Cookbook wmcs.ceph.set_cluster_in_maintenance

=== 2023-08-30 ===
* 13:07 wm-bot2: dcaro testing stuff

=== 2023-08-28 ===
* 15:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:05 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:05 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:04 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:03 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:02 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] ([[phab:T345084|T345084]]) - cookbook ran by root@cloudcumin1001
* 15:01 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-23 ===
* 16:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 16:10 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:10 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:09 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 16:08 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:08 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-15 ===
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 19:32 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 19:32 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 16:01 andrewbogott: rebooting cloudvirt2001-dev in an attempt to figure out what's happening with bastions
* 15:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 15:42 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 15:42 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:41 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:40 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 15:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
* 12:39 dcaro: removed some logs from the cloudmetrics1003:/var/log/carbon/ directory and stopped the carbon processes (they were crashing and filling up the disk with logs)

=== 2023-08-14 ===
* 22:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
* 22:11 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:11 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
* 22:10 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
* 22:09 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:08 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
* 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

=== 2023-08-07 ===
* 09:39 taavi: cloud vps graphite service was disabled: https://wikitech.wikimedia.org/wiki/News/2023_Cloud_VPS_metrics_changes [[phab:T326266|T326266]]

=== 2023-08-03 ===
* 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0)
* 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.do_log_msg
* 13:07 wm-bot2: Test SAL log ([[phab:T341793|T341793]]) - cookbook ran by root@cloudcumin1001

=== 2023-07-31 ===
* 16:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0)
* 16:16 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 16:15 fnegri@cloudcumin1001: START - Cookbook wmcs.do_log_msg
* 15:33 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:42 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:35 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 13:55 andrewbogott: recreating the codfw1dev galera cluster according to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Galera -- mariadb is stopped (and won't start) on all three cloudcontrol nodes
* 13:39 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001

=== 2023-07-27 ===
* 10:52 arturo: adding cloud-private subnet to cloudnet1005/1006 hosts in eqiad1 ([[phab:T342619|T342619]])

=== 2023-07-19 ===
* 14:02 wm-bot2: Draining cloudvirt2002-dev.codfw.wmnet ([[phab:T335840|T335840]]) - cookbook ran by raymond@ubuntu
* 14:02 wm-bot2: Safe rebooting cloudvirt2002-dev.codfw.wmnet ([[phab:T335840|T335840]]) - cookbook ran by raymond@ubuntu

=== 2023-07-17 ===
* 15:55 arturo: cloudcontrol1005 was shutdown earlier today ([[phab:T341495|T341495]])
* 15:35 arturo: [codfw1dev] cloudweb2002-dev up and running after reracking ([[phab:T327919|T327919]])
* 15:11 arturo: [codfw1dev] powered off cloudweb2002-dev for reracking ([[phab:T327919|T327919]])
* 12:45 taavi: removing diamond from remaining buster instances [[phab:T317032|T317032]]

=== 2023-07-13 ===
* 10:48 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by dcaro@urcuchillay
* 10:48 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:47 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:46 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:45 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:44 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:43 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:42 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:38 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
* 10:33 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay

=== 2023-07-07 ===
* 10:28 taavi: backfilling &lt;nowiki&gt;{&lt;/nowiki&gt;project&lt;nowiki&gt;}&lt;/nowiki&gt;.wmcloud.org and other currently-named DNS zones to projects that don't have them

=== 2023-07-06 ===
* 17:02 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:00 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:00 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-07-04 ===
* 14:08 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001
* 14:07 wm-bot2: Test SAL log ([[phab:T325756|T325756]]) - cookbook ran by root@cloudcumin1001

=== 2023-06-30 ===
* 21:37 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-06-29 ===
* 19:49 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:48 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:47 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:34 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-06-26 ===
* 08:46 arturo: [codfw1dev] manually start mariadb service @ cloudinfra-db-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud

=== 2023-06-24 ===
* 19:21 wm-bot2: Created new flavor: g3.cores16.ram34.disk20 (id:7dd33202-32c3-4bc7-b2d4-{{Gerrit|10c2ebe7e5c5}}) - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Created new flavor: g3.cores16.ram34816.disk20.admin (id:d76925a8-b58b-489e-a3d9-{{Gerrit|c68c7744b551}}) - cookbook ran by andrew@bullseye

=== 2023-06-23 ===
* 16:45 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:45 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:44 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:43 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:41 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 13:59 andrewbogott: rebooting every VM in codfw1dev

=== 2023-06-13 ===
* 15:35 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:35 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:34 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:32 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:31 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 14:58 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye

=== 2023-06-12 ===
* 19:37 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:37 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 11:57 arturo: [codfw1dev] refresh various occurrences of old FQDNs in instance puppet via horizon ([[phab:T324992|T324992]])

=== 2023-06-08 ===
* 16:56 andrewbogott: deleting all Stretch base images from glance
* 16:45 andrewbogott: updated the bullseye image with https://cloud.debian.org/images/cloud/bullseye/20230601-1398/debian-11-genericcloud-amd64-20230601-1398.tar.xz
* 12:17 wm-bot2: Drained cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:17 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 02920314-1efe-4934-ad81-{{Gerrit|d2a6cf2e17ab}}, use this to unset) ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:16 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:06 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 769349bf-465f-4f0c-a8f3-{{Gerrit|f2423631ba7e}}, use this to unset) ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T334644|T334644]]) - cookbook ran by dcaro@vulcanus

=== 2023-06-07 ===
* 10:06 dcaro: upgraded ruby2.5 to latest fixed version on all buster VMs
* 08:10 dcaro: downgrading ruby2.5 to previous backport on all buster VMs

=== 2023-06-06 ===
* 19:09 andrewbogott: also increased RAM and secgroup-rule quota for Trove  [[phab:T337882|T337882]]
* 19:06 andrewbogott: increased trove secgroups, instances, volumes quotas from 40 to 100.  Trove is too popular!  [[phab:T337882|T337882]]
* 18:31 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 18:31 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:30 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:29 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:28 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:27 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:26 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 18:26 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:55 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:54 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:42 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:42 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
* 17:41 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye

=== 2023-06-05 ===
* 09:41 arturo: [codfw1dev] rebooting bastion-codfw1dev-02 (no IP address in the main interface) [[phab:T336963|T336963]]

=== 2023-06-02 ===
* 14:40 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by arturo@nostromo
* 12:59 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 12:58 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:58 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:44 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 12:44 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 12:43 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-26 ===
* 16:03 andrewbogott: &quot;maintain-views --all-databases --replace-all&quot; on clouddb1021 for [[phab:T337446|T337446]]

=== 2023-05-22 ===
* 19:54 andrewbogott: deleting project 'citelearn' as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge#SHUTDOWN_citelearn

=== 2023-05-21 ===
* 23:29 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 23:28 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 23:27 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:45 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:44 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:44 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-19 ===
* 14:53 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:53 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:52 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-18 ===
* 21:39 andrewbogott: deleting obsolete roles '4d8cad783d6342efa8414d7d36fbc034 {{!}} projectadmin_renamed_for_[[phab:T330759|T330759]]' and 'f473273fac7146b3bdbf22e5d4504f95 {{!}} user_renamed_for_[[phab:T330759|T330759]]' on eqiad1. State pre-deletion is dumped to /root/allassignmentspredeletion.txt  on cloudcontrol1007.
* 21:35 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:35 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 21:34 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 21:33 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:20 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 19:18 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:12 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:11 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:10 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Restarting openstack services on cloudcontrol2001-dev@local1: ['cinder-volume'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:21 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:20 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-15 ===
* 14:28 wm-bot2: Drained cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:28 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: 96ce2ed0-3aff-4d04-be0b-{{Gerrit|e16513070617}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:27 wm-bot2: Draining cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:23 wm-bot2: Drained cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 110176f8-04d5-4110-bb7d-{{Gerrit|1ab272bd8be2}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:16 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: c6e92e13-49f4-4db3-8a13-{{Gerrit|8692ccfd3bc9}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:12 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:12 wm-bot2: Drained cloudvirt1035.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 14:11 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: fa730dec-848f-45fb-9eda-{{Gerrit|e74bd874c5c9}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:10 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1006: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1007: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 14:06 wm-bot2: Restarting openstack services on cloudcontrol1005: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
* 14:01 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: eb1cfac0-d481-4baa-b9cd-{{Gerrit|15e5fbcef495}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:00 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:58 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: 3e6c3ff3-7d55-4777-9032-{{Gerrit|b867a257eced}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:57 wm-bot2: Draining cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:53 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: 0693664a-df78-417e-ba34-{{Gerrit|590e5a0a9981}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:52 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:49 wm-bot2: Set cloudvirt cloudvirt1035.eqiad.wmnet maintenance (downtime id: e6929ab8-4bc3-4186-817b-{{Gerrit|9b53dbd597c6}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:48 wm-bot2: Draining cloudvirt1035.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:40 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:39 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:38 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:37 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 andrewbogott: restarting nova services in eqiad1, trying to free up db connections
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 13:33 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: fef43d73-6fd4-4dde-a0ac-{{Gerrit|95fd69a9b0c1}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:32 wm-bot2: Draining cloudvirt1034.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 5f867662-e824-498c-a715-{{Gerrit|2e2ad50f0bb5}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: 0f515150-1313-41d4-a5f6-{{Gerrit|9bc00ce9b245}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Draining cloudvirt1033.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Drained cloudvirt1032.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot2: Set cloudvirt cloudvirt1032.eqiad.wmnet maintenance (downtime id: e826adc3-addd-44d8-b39e-{{Gerrit|ae7bd2df1e60}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:49 wm-bot2: Draining cloudvirt1032.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:49 wm-bot2: Drained cloudvirt1031.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:14 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 4154c818-744c-4d84-9883-{{Gerrit|cae7a5826ed5}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:13 wm-bot2: Draining cloudvirt1027.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:13 wm-bot2: Drained cloudvirt1026.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:04 wm-bot2: Set cloudvirt cloudvirt1026.eqiad.wmnet maintenance (downtime id: cdfc3d01-ec1e-483c-9a38-{{Gerrit|834193e487ff}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Draining cloudvirt1026.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Drained cloudvirt1025.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 11:51 wm-bot2: Set cloudvirt cloudvirt1025.eqiad.wmnet maintenance (downtime id: 6a56757d-35de-499e-8209-{{Gerrit|728bcf62a22a}}, use this to unset) ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus
* 11:50 wm-bot2: Draining cloudvirt1025.eqiad.wmnet ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus

=== 2023-05-12 ===
* 17:52 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-05-10 ===
* 16:09 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:09 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:08 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:07 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:06 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:05 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:04 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 16:00 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:59 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:58 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:57 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:56 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:55 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:54 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:53 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 andrewbogott: running &quot;cookbook -c ~/.config/spicerack/cookbook_config.yaml wmcs.openstack.restart_openstack --cluster-name eqiad1 --all&quot; to pick up changes for testing [[phab:T336379|T336379]]
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 15:52 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 03:05 andrewbogott: &quot;systemctl restart puppet-enc&quot; on enc-1.cloudinfra.eqiad1.wikimedia.cloud.  Seems to have crashed.

=== 2023-05-05 ===
* 16:07 wm-bot2: Drained cloudvirt1024.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:03 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 95995009-09d6-496e-8cd2-{{Gerrit|0cfac93d3cf7}}, use this to unset) ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:02 wm-bot2: Draining cloudvirt1024.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 16:01 wm-bot2: Drained cloudvirt1023.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:51 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 53c46cae-00af-4664-97ff-{{Gerrit|266b393335bb}}, use this to unset) ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:50 wm-bot2: Draining cloudvirt1023.eqiad.wmnet ([[phab:T336064|T336064]]) - cookbook ran by andrew@bullseye
* 15:49 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 528ea4f6-8088-475e-937f-{{Gerrit|098ffba861b6}}, use this to unset) - cookbook ran by andrew@bullseye
* 15:47 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 3ef85b5e-d9d9-4b24-901b-{{Gerrit|a3058a7d0615}}, use this to unset) - cookbook ran by andrew@bullseye
* 15:44 andrewbogott: moved cloudvirt1023 and cloudvirt1024 from 'ceph' aggregate to 'maintenance' aggregate, prep for decom [[phab:T336064|T336064]]
* 15:44 andrewbogott: moved cloudvirt1028 from 'localdisk' aggregate to 'maintenance' aggregate. Nothing new should be scheduled here, local storage should now move to cloudvirtlocal100x
* 15:41 andrewbogott: moved cloudvirt1055 and cloudvirt1056 from 'spare' to 'ceph' aggregate.  Prep for removing two obsolete cloudvirts, 1023 and 1024.  [[phab:T336064|T336064]]

=== 2023-05-04 ===
* 22:49 andrewbogott: removed fullstack-* puppet reports on puppetmaster-02.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud and cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs to free up disk space

=== 2023-05-02 ===
* 13:01 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:31 wm-bot2: Destroying OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 12:30 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Added 1 new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Added OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 11:53 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus

=== 2023-05-01 ===
* 17:09 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 17:09 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 17:08 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 17:08 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 15:22 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
* 13:53 taavi: running wmcs-novastats-puppetleaks in real mode [[phab:T334127|T334127]]
* 13:51 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus

=== 2023-04-18 ===
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1007: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1006: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudcontrol1005: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
* 22:52 wm-bot2: Restarting openstack services on cloudvirt1056: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1019: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1060: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1023: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1038: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1036: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1039: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1050: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1061: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1028: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1020: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1052: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1040: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1043: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1026: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1030: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1048: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1025: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Restarting openstack services on cloudvirt1035: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1055: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1042: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1059: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1057: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1032: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1029: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1044: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1047: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1034: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1058: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1045: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1041: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1031: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1046: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1027: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Restarting openstack services on cloudvirt1054: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1033: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1037: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1051: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1053: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:49 wm-bot2: Restarting openstack services on cloudvirt1049: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:48 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:47 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1019: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1020: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:45 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:44 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:41 andrewbogott: resetting rabbitmq on cloudrabbit1003 due to splitbrain
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:40 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:39 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:31 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1019: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1020: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:29 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye

=== 2023-04-17 ===
* 08:49 wm-bot2: Increased quotas by 9 cores, 1 instances, 16 ram ([[phab:T334695|T334695]]) - cookbook ran by dcaro@vulcanus

=== 2023-04-06 ===
* 17:03 andrewbogott: running wmcs-wikireplica-dns on cloudcontrol1005 to update tools-db dns entries

=== 2023-04-04 ===
* 17:23 andrewbogott: resetting all three rabbitmq nodes and restarting all openstack services as per https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Rabbitmq#Resetting_the_HA_setup

=== 2023-03-28 ===
* 13:56 andrewbogott: depooling cloudweb1003 before switch upgrade
* 10:58 dhinus: disabled tool &quot;wb&quot; by clicking the disable button at https://toolsadmin.wikimedia.org/tools/id/wb [[phab:T328693|T328693]]
* 08:34 arturo: cleanup neutron agents for cloudvirt1021/1022 (decom)
* 08:32 arturo: cleanup neutron agents for cloudvirt1017 (decom)

=== 2023-03-27 ===
* 14:30 wm-bot2: Drained cloudvirt1024.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:19 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 3f43d3ca-696c-4d3c-8d5b-{{Gerrit|e57984f0eb86}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:18 wm-bot2: Draining cloudvirt1024.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:17 wm-bot2: Drained cloudvirt1023.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:08 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: f4767781-ea26-453b-9521-{{Gerrit|847a8340a249}}, use this to unset) - cookbook ran by andrew@bullseye
* 14:07 wm-bot2: Draining cloudvirt1023.eqiad.wmnet - cookbook ran by andrew@bullseye
* 14:04 wm-bot2: Drained cloudvirt1022.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:55 wm-bot2: Set cloudvirt cloudvirt1022.eqiad.wmnet maintenance (downtime id: 0e794cfe-5896-46c1-842d-{{Gerrit|c34719140d4f}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:54 wm-bot2: Draining cloudvirt1022.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:54 wm-bot2: Drained cloudvirt1021.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:48 wm-bot2: Set cloudvirt cloudvirt1021.eqiad.wmnet maintenance (downtime id: e7a904ca-003d-450a-ad85-{{Gerrit|886fb80dfc41}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:47 wm-bot2: Draining cloudvirt1021.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:46 wm-bot2: Drained cloudvirt1017.eqiad.wmnet - cookbook ran by andrew@bullseye
* 13:36 wm-bot2: Set cloudvirt cloudvirt1017.eqiad.wmnet maintenance (downtime id: 0ff0090f-26c3-4278-a9c9-{{Gerrit|cce518559408}}, use this to unset) - cookbook ran by andrew@bullseye
* 13:35 wm-bot2: Draining cloudvirt1017.eqiad.wmnet - cookbook ran by andrew@bullseye

=== 2023-03-22 ===
* 12:41 taavi: delete wmde-templates-alpha project [[phab:T332773|T332773]]

=== 2023-03-08 ===
* 21:45 bd808: maintain-kubeusers container in CrashLoopBackoff, investigating
* 13:49 dcaro: stopping puppet on labostre1004 to debug maintain-dbusers

=== 2023-03-07 ===
* 16:06 andrewbogott: updated application credential roles, replacing 'user' with 'reader' and 'projectadmin' with 'member':  update application_credential_role set role_id='f75a3c410bca4e96a1cf6ac103b0ccaf' where role_id='f473273fac7146b3bdbf22e5d4504f95'  and update application_credential_role set role_id='38676f30eaeb44518bf7e144a73c8da6' where role_id='4d8cad783d6342efa8414d7d36fbc034'
* 10:11 dcaro: there was a little unavailability for some VMs while ceph was starting to rebalance things, but it seems stable and moving data around ([[phab:T331141|T331141]])
* 09:37 dcaro: Changing ceph crush map to allow rack HA on eqiad1 cluster ([[phab:T331141|T331141]])

=== 2023-03-03 ===
* 12:12 arturo: installing haproxy updates ([[phab:T331119|T331119]])

=== 2023-03-02 ===
* 16:17 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1010.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Added 1 new OSDs ['cloudcephosd1010.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Added OSD cloudcephosd1010.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot2: Finished rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:10 wm-bot2: Rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Adding OSD cloudcephosd1010.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Adding new OSDs ['cloudcephosd1010.eqiad.wmnet'] to the cluster ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 10:43 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1005.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:44 wm-bot2: Added 1 new OSDs ['cloudcephosd1005.eqiad.wmnet'] ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:44 wm-bot2: Added OSD cloudcephosd1005.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:36 wm-bot2: Finished rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:33 wm-bot2: Rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Adding OSD cloudcephosd1005.eqiad.wmnet... (1/1) ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Adding new OSDs ['cloudcephosd1005.eqiad.wmnet'] to the cluster ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-28 ===
* 22:47 andrewbogott: adding new 'member' role assignment to every user/project pair that currently has the 'user' assignment.  [[phab:T330759|T330759]]
* 09:47 wm-bot2: Depooled and destroyed OSD daemons [79, 78, 77, 76, 75, 74, 73, 72] and removed the OSD host cloudcephosd1010 from the CRUSH map. ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:46 wm-bot2: Destroying OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:28 wm-bot2: Depooling OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot2: Depooling OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:12 wm-bot2: Depooled and destroyed OSD daemons [39, 38, 37, 36, 35, 34, 33, 32] and removed the OSD host cloudcephosd1005 from the CRUSH map. ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 09:11 wm-bot2: Destroying OSDs with ids in [39, 38, 37, 36, 35, 34, 33, 32] on cloudcephosd1005 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus
* 08:55 wm-bot2: Depooling OSDs with ids in [39, 38, 37, 36, 35, 34, 33, 32] on cloudcephosd1005 from eqiad1 ([[phab:T329504|T329504]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-27 ===
* 21:01 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1004.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:53 wm-bot2: Added 1 new OSDs ['cloudcephosd1004.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:53 wm-bot2: Added OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:45 wm-bot2: Finished rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:42 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:41 wm-bot2: Adding OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:41 wm-bot2: Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Adding OSD cloudcephosd1004.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:38 wm-bot2: Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 18:14 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1003.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:13 wm-bot2: Added 1 new OSDs ['cloudcephosd1003.eqiad.wmnet'] ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:13 wm-bot2: Added OSD cloudcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:05 wm-bot2: Finished rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:02 wm-bot2: Rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding OSD cloudcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding new OSDs ['cloudcephosd1003.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:59 wm-bot2: Adding OSD coludcephosd1003.eqiad.wmnet... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:59 wm-bot2: Adding new OSDs ['coludcephosd1003.eqiad.wmnet'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:58 wm-bot2: Adding OSD coludcephosd1003... (1/1) ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:58 wm-bot2: Adding new OSDs ['coludcephosd1003'] to the cluster ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-22 ===
* 08:25 wm-bot2: Depooled and destroyed OSD daemons [31, 30, 29, 28, 27, 26, 25, 24] and removed the OSD host cloudcephosd1004 from the CRUSH map. ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:22 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:15 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 08:13 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 07:23 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 07:02 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-21 ===
* 20:27 andrewbogott: deleted 200 more orphaned VM images with wmcs-novastats-cephleaks
* 17:18 andrewbogott: shutting down postgres on clouddb1004/1003, then shutting down the vms
* 15:50 wm-bot2: Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 15:48 wm-bot2: Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot2: Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus
* 14:00 wm-bot2: Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 ([[phab:T329502|T329502]]) - cookbook ran by dcaro@vulcanus

=== 2023-02-16 ===
* 19:14 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1002.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:55 dcaro: Manually zapped /dev/sdc on cloudcephosd1002, probably a leftover drive since the beginning (or during the reimage the drives changed names, and this one had leftovers from the previous OS) ([[phab:T329498|T329498]])
* 17:47 wm-bot2: Added 1 new OSDs ['cloudcephosd1002.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:47 wm-bot2: Added OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:42 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 17:42 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:50 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:50 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:01 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:00 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 16:00 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot2: Added 1 new OSDs ['cloudcephosd1001.eqiad.wmnet'] ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:29 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Destroying OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot2: Depooling OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:20 wm-bot2: Destroying OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:19 wm-bot2: Depooling OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:03 wm-bot2: Destroying OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 11:01 wm-bot2: Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 10:59 wm-bot2: Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 10:15 dcaro: purges osd daemons 48 and 40 from eqiad ceph cluster ([[phab:T329709|T329709]])

=== 2023-02-15 ===
* 14:53 andrewbogott: deleting another 100 leaked VM images with wmcs-novastats-cephleaks
* 13:39 wm-bot2: Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot2: Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:13 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:11 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:11 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot2: Destroying OSDs with id [[12345]] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
* 13:09 wm-bot2: Destroying OSDs with id ['12345'] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus

=== 2023-02-14 ===
* 13:17 andrewbogott: restarting all eqiad1 openstack services because that seems to sometimes help things *shrug*

=== 2023-02-13 ===
* 14:06 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 8fbf6bfd-eec1-4d81-8e0d-{{Gerrit|ea431d8411ee}} ([[phab:T329498|T329498]]) - cookbook ran by dcaro@vulcanus
* 13:32 taavi: re-enable puppet on labstore1004 [[phab:T329377|T329377]]

=== 2023-02-09 ===
* 21:17 andrewbogott: deleted 10% of leaked VM ceph images using wmcs-novastats-cephleaks (only 10% out of an abundance of caution)

=== 2023-02-08 ===
* 17:08 arturo: changing to cloudgw network setup, make VIPs /32 ([[phab:T295774|T295774]])

=== 2023-02-07 ===
* 11:26 arturo: [codfw1dev] testing network changes in cloudgw, expect unrealiable network ([[phab:T295774|T295774]])

=== 2023-02-04 ===
* 13:44 taavi: drop old columns from oathauth_users table on labtestwiki [[phab:T328131|T328131]]

=== 2023-02-03 ===
* 15:00 andrewbogott: restarted nova services in eqiad1 in an attempt to eke out another day or two of stability
* 14:13 taavi: attached GrapheSuppression developer account to wikitech

=== 2023-02-02 ===
* 13:14 dcaro_away: draining osd.48 from node cloudcephosd1001 ([[phab:T316544|T316544]])
* 12:57 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 7ac2b25a-d1bb-4789-8aa6-{{Gerrit|b9435b505349}} ([[phab:T316544|T316544]]) - cookbook ran by dcaro@vulcanus

=== 2023-01-30 ===
* 22:34 wm-bot2: Upgraded and rebooted host cloudrabbit1002.wikimedia.org - cookbook ran by andrew@bullseye
* 21:34 andrewbogott: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/884922 and upgrading rabbitmq nodes for [[phab:T328155|T328155]]

=== 2023-01-27 ===
* 20:08 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:22 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 15:25 andrewbogott: restarting openstack services in eqiad1, another attempt to address instability

=== 2023-01-26 ===
* 20:34 andrewbogott: shutting down mariadb on cloudbackup2001-dev, testing the waters for [[phab:T328079|T328079]]

=== 2023-01-22 ===
* 03:42 andrewbogott: reset eqiad1 rabbitmq in an attempt to resolve some mild instability

=== 2023-01-20 ===
* 15:26 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 15:26 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['f47a3d91-b270-4c90-acc8-d85075a6bf8e'], use this to unset) - cookbook ran by andrew@bullseye
* 13:15 arturo: reinstall python3-neutron (to reset manual patching) on all cloudnet nodes and patch it via puppet, then restart neutron-l3-agent by hand ([[phab:T327463|T327463]])
* 10:12 arturo: [codfw1dev] failover neutron-l3-agent between cloudnet2005-dev/cloudnet2006-dev a couple of times [[phab:T327463|T327463]]
* 02:17 andrewbogott: stopping neutron-l3-agent on cloudnet1005 because it's logging at a furious rate and about to fill the drive

=== 2023-01-19 ===
* 18:06 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 18:06 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['36d5af6a-7d8e-4d0c-831e-1bf05c255984'], use this to unset) - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
* 17:22 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['66ec4f04-d25b-4067-be9e-2fe12cb1d3ff'], use this to unset) - cookbook ran by andrew@bullseye

=== 2023-01-18 ===
* 22:32 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 347cb75e-215e-4b85-ae14-{{Gerrit|4ce1934c70c7}}, use this to unset) - cookbook ran by andrew@bullseye
* 22:01 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 9bf6212f-7fdb-4869-8190-{{Gerrit|b07387b2bc7e}}, use this to unset) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 11aec8ea-f443-41ae-b79a-{{Gerrit|5e1e3aa94546}}, use this to unset) - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:03 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:02 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 17:01 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:38 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:38 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 14:37 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
* 14:37 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

=== 2023-01-17 ===
* 19:32 wm-bot2: Upgraded and rebooted host cloudbackup2002.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:04 wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye
* 17:52 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 17:23 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye

=== 2023-01-13 ===
* 17:36 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:36 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:35 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
* 17:26 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
* 17:26 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:25 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
* 17:21 wm-bot2: Restarting openstack services: &lt;nowiki&gt;{&lt;/nowiki&gt;'cloudcontrol2001-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'], 'cloudcontrol2004-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'], 'cloudvirt2001-dev': ['nova-compute'], 'cloudvirt2002-dev': ['nova-compute'], 'cloudvirt2003-dev': ['nova-compute'], 'cloudcontrol2005-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata
* 10:41 arturo: restart backup_vm.service on cloudbackup1003/1004 to recover from a nova Unknown Error (HTTP 503)

=== 2023-01-12 ===
* 22:34 andrewbogott: updated the Bullseye base image with the upstream {{Gerrit|20221219}} build
* 14:12 wm-bot2: Created project checkuser-beta-wiki with default quotas. ([[phab:T326740|T326740]]) - cookbook ran by arturo@nostromo

=== 2023-01-06 ===
* 18:42 wm-bot2: Safe reboot of cloudvirt2003-dev.codfw.wmnet finished successfully - cookbook ran by andrew@bullseye
* 18:42 wm-bot2: unset cloudvirt2003-dev.codfw.wmnet maintenance (aggregates: ceph) - cookbook ran by andrew@bullseye
* 18:39 wm-bot2: Drained cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: b50d9d3a-4f1d-4522-b86f-{{Gerrit|722fc9c55c87}}, use this to unset) - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Safe rebooting cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Draining cloudvirt2003-dev.cdofw.wmnet - cookbook ran by andrew@bullseye
* 18:35 wm-bot2: Safe rebooting cloudvirt2003-dev.cdofw.wmnet - cookbook ran by andrew@bullseye
* 00:14 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 00:08 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 00:01 wm-bot2: Upgraded and rebooted host cloudnet2006-dev.codfw.wmnet - cookbook ran by andrew@bullseye

=== 2023-01-05 ===
* 23:54 wm-bot2: Upgraded and rebooted host cloudnet2005-dev.codfw.wmnet - cookbook ran by andrew@bullseye
* 23:38 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 23:25 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 23:13 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 22:18 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 22:10 andrewbogott: upgrading codfw1dev openstack to version 'zed'

=== 2023-01-04 ===
* 21:18 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org - cookbook ran by andrew@bullseye
* 21:11 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org - cookbook ran by andrew@bullseye
* 20:12 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org - cookbook ran by andrew@bullseye
* 20:04 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org - cookbook ran by andrew@bullseye
* 14:45 wm-bot2: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] - cookbook ran by fran@wmf3169
* 14:44 wm-bot2: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:41 wm-bot2: Rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:41 wm-bot2: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:38 wm-bot2: Rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:37 wm-bot2: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev - cookbook ran by fran@wmf3169

=== 2023-01-03 ===
* 22:11 wm-bot2: Upgraded and rebooted host cloudservices2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Upgraded and rebooted host cloudservices2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 15:25 taavi: restart designate-sink everywhere to pick up wmf-sink changes

=== 2022-12-25 ===
* 14:21 taavi: register developer account 'instance-puppet-user-dev' to update the codfw1dev instance-puppet repo without access to the eqiad1 repo [[phab:T318504|T318504]]

=== 2022-12-22 ===
* 15:16 dcaro: added submit rights for JenkinsBot on all cloud/* gerrit repos

=== 2022-12-21 ===
* 04:59 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:59 wm-bot2: Finished rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:35 wm-bot2: Rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:35 wm-bot2: Finished rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:12 wm-bot2: Rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 04:12 wm-bot2: Finished rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:48 wm-bot2: Rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:48 wm-bot2: Finished rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:24 wm-bot2: Rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:24 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:00 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 03:00 wm-bot2: Finished rebooting node cloudcephosd1024.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:36 wm-bot2: Rebooting node cloudcephosd1024.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:36 wm-bot2: Finished rebooting node cloudcephosd1023.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:12 wm-bot2: Rebooting node cloudcephosd1023.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 02:12 wm-bot2: Finished rebooting node cloudcephosd1022.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:47 wm-bot2: Rebooting node cloudcephosd1022.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:47 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:22 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 01:22 wm-bot2: Finished rebooting node cloudcephosd1020.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:58 wm-bot2: Rebooting node cloudcephosd1020.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:58 wm-bot2: Finished rebooting node cloudcephosd1019.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:34 wm-bot2: Rebooting node cloudcephosd1019.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:34 wm-bot2: Finished rebooting node cloudcephosd1018.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:09 wm-bot2: Rebooting node cloudcephosd1018.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 00:09 wm-bot2: Finished rebooting node cloudcephosd1017.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye

=== 2022-12-20 ===
* 23:45 wm-bot2: Rebooting node cloudcephosd1017.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:45 wm-bot2: Finished rebooting node cloudcephosd1016.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:21 wm-bot2: Rebooting node cloudcephosd1016.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 23:21 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:56 wm-bot2: Rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:55 wm-bot2: Finished rebooting node cloudcephosd1014.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Rebooting node cloudcephosd1014.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:30 wm-bot2: Finished rebooting node cloudcephosd1013.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:08 andrewbogott: restarting openstack services with wmcs.openstack.restart_openstack due to miscellaneous trove failures
* 22:06 wm-bot2: Rebooting node cloudcephosd1013.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 22:06 wm-bot2: Finished rebooting node cloudcephosd1012.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Rebooting node cloudcephosd1012.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:40 wm-bot2: Finished rebooting node cloudcephosd1011.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Rebooting node cloudcephosd1011.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Finished rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Rebooting node cloudcephosd1010.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Finished rebooting node cloudcephosd1009.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:26 wm-bot2: Rebooting node cloudcephosd1009.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:26 wm-bot2: Finished rebooting node cloudcephosd1008.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:01 wm-bot2: Rebooting node cloudcephosd1008.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 20:01 wm-bot2: Finished rebooting node cloudcephosd1007.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Rebooting node cloudcephosd1007.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:35 wm-bot2: Finished rebooting node cloudcephosd1006.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Rebooting node cloudcephosd1006.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 19:10 wm-bot2: Finished rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:45 wm-bot2: Rebooting node cloudcephosd1005.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:45 wm-bot2: Finished rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:20 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 18:20 wm-bot2: Finished rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:56 wm-bot2: Rebooting node cloudcephosd1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:56 wm-bot2: Finished rebooting node cloudcephosd1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Rebooting node cloudcephosd1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:33 wm-bot2: Finished rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:07 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 17:07 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 16:49 wm-bot2: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:48 wm-bot2: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:46 wm-bot2: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:46 wm-bot2: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:42 wm-bot2: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:40 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 16:39 wm-bot2: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 16:39 wm-bot2: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:41 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:41 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
* 15:40 wm-bot2: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:40 wm-bot2: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:37 wm-bot2: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:37 wm-bot2: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:33 wm-bot2: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:30 wm-bot2: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:30 wm-bot2: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:29 wm-bot2: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:29 wm-bot2: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:25 wm-bot2: Rebooting node cloudcephmon2006-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:25 wm-bot2: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Rebooting node cloudcephmon2005-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:22 wm-bot2: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:19 wm-bot2: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:19 wm-bot2: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:18 wm-bot2: Finished rebooting the nodes ['cloudcephmon1001', 'cloudcephmon1002', 'cloudcephmon1003'] ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:18 wm-bot2: Finished rebooting node cloudcephmon1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:15 wm-bot2: Rebooting node cloudcephmon1003.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:15 wm-bot2: Finished rebooting node cloudcephmon1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:11 wm-bot2: Finished rebooting node cloudcephmon1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:08 wm-bot2: Rebooting node cloudcephmon1001.eqiad.wmnet ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye
* 15:08 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 ([[phab:T325132|T325132]]) - cookbook ran by andrew@bullseye

=== 2022-12-17 ===
* 07:50 taavi: deleted project packagist-mirror per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge#packagist-mirror

=== 2022-12-16 ===
* 19:36 volans: restarted sshd twice on bastion-restricted-eqiad1-02 to debug SSH connections for [[phab:T319401|T319401]]
* 08:46 dcaro: restart designate-sink on both cloudservice hosts ([[phab:T322279|T322279]])
* 08:45 dcaro: restart designate-sink on both cloudservice hosts

=== 2022-12-07 ===
* 22:07 andrewbogott: systemctl restart libvirt-guests.service  on cloudvirt1019 to get ceph/rbd working on VMS on this hypervisor

=== 2022-12-03 ===
* 19:24 taavi: restart designate-sink on both cloudservices hosts

=== 2022-11-30 ===
* 20:03 andrewbogott: changing all rabbitmq queues to quorum queues.  Will be noisy!  [[phab:T318816|T318816]]
* 02:54 wm-bot2: Upgraded and rebooted host cloudbackup2002.codfw.wmnet - cookbook ran by andrew@bullseye

=== 2022-11-28 ===
* 13:00 wm-bot2: unset cloudvirt1043.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 10:28 wm-bot2: Drained cloudvirt1043.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:19 wm-bot2: Set cloudvirt cloudvirt1043.eqiad.wmnet maintenance (downtime id: bb94dd24-fef9-4c9c-8f79-{{Gerrit|b6e15023ce69}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:18 wm-bot2: Draining cloudvirt1043.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-25 ===
* 10:54 wm-bot2: deleted VM canary2001-dev-2 from cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 10:54 wm-bot2: created VM canary2001-dev-3 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 10:53 wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:5b2ca632-2ea0-4007-9b40-{{Gerrit|4f84f8e2428b}}) - cookbook ran by arturo@nostromo
* 10:46 wm-bot2: Created new flavor: g3.cores1.ram1.disk20.admin (id:fecbd56d-0969-45f3-80fd-{{Gerrit|2b463a5b6270}}) - cookbook ran by arturo@nostromo

=== 2022-11-24 ===
* 16:53 wm-bot2: deleted VM canary2001-dev-1 from cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:53 wm-bot2: created VM canary2001-dev-2 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2003-dev-1 in cloudvirt2003-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2002-dev-1 in cloudvirt2002-dev - cookbook ran by arturo@nostromo
* 16:42 wm-bot2: created VM canary2001-dev-1 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
* 16:36 wm-bot2: Created new flavor: cloudvirt-canary-ceph (id:0d06701a-2845-4298-b2b4-{{Gerrit|fabf8b1ddcbb}}) - cookbook ran by arturo@nostromo
* 13:03 wm-bot2: unset cloudvirt1044.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 11:51 wm-bot2: Drained cloudvirt1044.eqiad.wmnet - cookbook ran by arturo@nostromo
* 11:37 wm-bot2: Set cloudvirt cloudvirt1044.eqiad.wmnet maintenance (downtime id: 10076ecc-0f94-4d56-9bbd-{{Gerrit|bebb48bdc126}}, use this to unset) - cookbook ran by arturo@nostromo
* 11:35 wm-bot2: Draining cloudvirt1044.eqiad.wmnet - cookbook ran by arturo@nostromo
* 10:16 dcaro: removed ip6 dns name entry from nb for coluddb* ([[phab:T323550|T323550]])
* 09:53 dcaro: removed ip6 dns entry from nb for coluddb1013 ([[phab:T323550|T323550]])

=== 2022-11-23 ===
* 15:00 wm-bot2: unset cloudvirt1045.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 13:46 wm-bot2: Drained cloudvirt1045.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:35 wm-bot2: Set cloudvirt cloudvirt1045.eqiad.wmnet maintenance (downtime id: 2386b468-0f21-4ecb-91e2-{{Gerrit|e19ace66881d}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:34 wm-bot2: Draining cloudvirt1045.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:20 wm-bot2: unset cloudvirt1046.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 12:17 wm-bot2: Drained cloudvirt1046.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:04 wm-bot2: Set cloudvirt cloudvirt1046.eqiad.wmnet maintenance (downtime id: 6291d38e-c04c-4aa0-88da-{{Gerrit|2b329874a9b9}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:03 wm-bot2: Draining cloudvirt1046.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:02 wm-bot2: unset cloudvirt1047.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 11:32 arturo: [codfw1dev] created project cloudvirt-canary
* 10:13 wm-bot2: Drained cloudvirt1047.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:01 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 2c7ccc17-2be3-427d-aaec-{{Gerrit|57fadca0de5b}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:00 wm-bot2: Draining cloudvirt1047.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-22 ===
* 13:26 wm-bot2: unset cloudvirt1048.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 12:15 wm-bot2: unset cloudvirt1049.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
* 10:58 wm-bot2: Drained cloudvirt1049.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:37 wm-bot2: Set cloudvirt cloudvirt1049.eqiad.wmnet maintenance (downtime id: 3ae0a20d-3cf0-4eba-bea6-{{Gerrit|45aa61d8ad00}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:36 wm-bot2: Draining cloudvirt1049.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 10:21 wm-bot2: Unset cloudvirt cloudvirt1050.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo

=== 2022-11-21 ===
* 16:19 wm-bot2: Drained cloudvirt1050.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:11 wm-bot2: Set cloudvirt cloudvirt1050.eqiad.wmnet maintenance (downtime id: 0b154a93-a9d3-4ac7-bcfc-{{Gerrit|b67c49abe97b}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:09 wm-bot2: Draining cloudvirt1050.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:07 wm-bot2: Unset cloudvirt cloudvirt1051.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 15:18 wm-bot2: Drained cloudvirt1051.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 15:02 wm-bot2: Set cloudvirt cloudvirt1051.eqiad.wmnet maintenance (downtime id: 1de1174b-ec46-47a3-911c-{{Gerrit|b5808ce37028}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 15:00 wm-bot2: Draining cloudvirt1051.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 14:57 wm-bot2: Unset cloudvirt cloudvirt1052.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 12:53 wm-bot2: Drained cloudvirt1052.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:30 wm-bot2: Set cloudvirt cloudvirt1052.eqiad.wmnet maintenance (downtime id: 30db8a9b-08db-456f-8106-{{Gerrit|53188ff5f989}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:29 wm-bot2: Draining cloudvirt1052.eqiad.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 12:15 wm-bot2: Unset cloudvirt cloudvirt1053.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
* 10:13 arturo: drained cloudvirt1053 in preparation for reimage (was spare anyway)

=== 2022-11-18 ===
* 13:37 arturo: [codfw1dev] reimaged cloudvirt2001-dev and cloudvirt2002-dev
* 11:36 wm-bot2: Set cloudvirt cloudvirt2001-dev.codfw.wmnet maintenance (downtime id: 2fb9df34-fc2d-45b9-b21f-{{Gerrit|c0ec09008b92}}, use this to unset) - cookbook ran by arturo@nostromo
* 11:36 wm-bot2: Draining cloudvirt2001-dev.codfw.wmnet - cookbook ran by arturo@nostromo

=== 2022-11-16 ===
* 20:07 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:56 wm-bot2: Upgraded and rebooted host cloudservices2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 19:16 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 18:59 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 12:52 arturo: failovered cloudgw1002 into cloudgw1001 for reimage, IRC bots were briefly disconnected

=== 2022-11-14 ===
* 20:22 wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye
* 20:07 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
* 19:42 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye
* 19:28 andrewbogott: beginning OpenStack upgrade in eqiad1 -- [[phab:T305828|T305828]]
* 19:22 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 19:14 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
* 19:08 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 12:52 arturo: cleanup old network vlan interface names from /etc/network/interfaces in cloudnet1005/1006

=== 2022-11-11 ===
* 13:24 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: edad3915-b7c6-4b23-bb9c-{{Gerrit|ab13b04a41c5}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:23 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:17 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: a09a9868-8aae-4dc0-8510-{{Gerrit|a3923b703060}}, use this to unset) ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 13:16 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-10 ===
* 16:19 wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: 346013ec-ce4e-497e-ad65-{{Gerrit|2d215b14998c}}, use this to unset). ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:18 wm-bot2: Draining 'cloudvirt2002-dev.codfw.wmnet'. ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo
* 16:13 wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: a40a8487-22ee-4bc6-bbe4-{{Gerrit|694a615e3bf5}}, use this to unset). ([[phab:T319184|T319184]]) - cookbook ran by arturo@nostromo

=== 2022-11-08 ===
* 11:17 taavi: backfilling security groups for metricsinfra access on all projects [[phab:T288108|T288108]]

=== 2022-11-07 ===
* 21:01 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 20:50 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 20:40 andrewbogott: upgrading eqiad1 designate to version 'yoga'

=== 2022-11-04 ===
* 17:57 andrewbogott: removing cinderv2 API endpoints from keystone catalog; this is deprecated and removed in Yoga.  prep for [[phab:T305828|T305828]]

=== 2022-11-03 ===
* 19:24 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 19:19 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 00:08 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye

=== 2022-11-02 ===
* 23:39 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:38 wm-bot2: Upgraded and rebooted host cloudnet2006-dev.codfw.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:29 wm-bot2: Upgraded and rebooted host cloudnet2005-dev.codfw.wmnet ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:13 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 23:00 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org ([[phab:T305828|T305828]]) - cookbook ran by andrew@bullseye
* 22:54 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
* 20:29 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye

=== 2022-10-31 ===
* 13:09 arturo: restart keepalived on all 4 cloudgw servers to run them with `-D` in /etc/default/keepalived to further debug [[phab:T320975|T320975]]

=== 2022-10-26 ===
* 16:18 wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:bf48880d-0c1b-4c2a-8e8b-{{Gerrit|778d28b16561}}) ([[phab:T319446|T319446]]) - cookbook ran by dcaro@vulcanus
* 09:34 taavi: running wmcs-puppetcertleaks in delete mode
* 09:09 taavi: running wmcs-novastats-dnsleaks in delete mode

=== 2022-10-25 ===
* 16:03 arturo: [codfw1dev] [[phab:T321220|T321220]] root@cloudcontrol2001-dev:~# openstack subnet create magnum --no-dhcp --network 57017d7c-3817-429a-8aa3-{{Gerrit|b028de82cdcc}} --ip-version 4 --gateway auto --subnet-range 192.168.0.0/24
* 14:38 arturo: [codfw1dev] restart neutron-l3-agent in cloudnet2005-dev, it was dead after rabbit connectivity problems

=== 2022-10-24 ===
* 18:50 wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Finished rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:47 wm-bot2: Rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:47 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by andrew@bullseye
* 18:38 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by andrew@bullseye
* 18:21 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by andrew@bullseye
* 18:21 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl

=== 2022-10-20 ===
* 23:23 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:23 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:20 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:20 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:19 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:16 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:07 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:07 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:06 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:06 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 101c41d1-d65d-4088-b6d2-{{Gerrit|eac859e45ef8}}, use this to unset). - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 23:03 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 23:01 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:57 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:51 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 7197ce34-cc57-4677-a1a9-{{Gerrit|05e20cd0dd80}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:50 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:46 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:38 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 68e4cc1d-9def-444e-84a7-{{Gerrit|21b0b5adfa72}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 73388c1e-369b-40af-a2c1-{{Gerrit|96936128c324}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 12feeca2-ea59-48e9-9235-{{Gerrit|1cecf5b384cd}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:37 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:36 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:35 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 1b34587f-2770-4d2e-bb31-{{Gerrit|c8ad14632d39}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:34 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:33 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:32 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:28 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:24 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:13 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: e75b00eb-7f58-4821-8139-{{Gerrit|3dfc6e97a92a}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 9151511e-bcc0-4367-a976-{{Gerrit|7cff060308aa}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 241c0bd9-c3cf-40e4-95dc-{{Gerrit|9b51d9823fe9}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 7a53c274-1d5f-4c94-9a0d-{{Gerrit|721c3e8f7239}}, use this to unset). - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:12 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:11 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 22:10 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 22:10 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 22:06 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:58 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:55 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:54 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:54 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:51 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 0b75b44c-1efe-4edf-8f5e-{{Gerrit|42a67e8d3b13}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: aeb9c3a6-961a-4873-85d4-{{Gerrit|929248aebb8b}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:38 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: eb14cc04-3293-4cf8-a46f-{{Gerrit|1f87d8b0bcc4}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 84a3cd99-2643-495a-86b6-{{Gerrit|7ae80b11a30d}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:37 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:36 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:32 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:32 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:28 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:28 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 01a0b69c-2e27-4331-9635-{{Gerrit|403a668dac29}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:27 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:27 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:23 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: ea29f65f-6d86-4568-8db9-{{Gerrit|a85faa827447}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:22 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:22 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:19 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:18 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 21:18 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 21:16 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:15 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:14 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: d6b6bb4d-c7c0-4bd2-9a02-{{Gerrit|dee9a5ec6a3e}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 4e109149-7d67-403f-8b21-{{Gerrit|f829235ea491}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: 341f4b8c-6d6e-4190-9f2d-{{Gerrit|a1a1483abadd}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:01 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: f76e8f14-920e-4d51-a44e-{{Gerrit|321a55dcb0c5}}, use this to unset). - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 21:00 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:56 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:56 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:54 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:35 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:34 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 0b0d8090-4d31-4d43-8545-{{Gerrit|c25209e2ef58}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:31 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 3d928554-a79c-419a-bcc4-{{Gerrit|c8d63791d8e7}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 3d3c8aa5-45f7-429e-a9d7-{{Gerrit|b5181b29dc13}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: b9027020-8a90-4c40-b0e6-{{Gerrit|6c8767f87917}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: f0d60fab-64e9-4da2-826c-{{Gerrit|229d31d0fbc2}}, use this to unset). - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:21 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:19 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:18 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:16 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 20:15 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 20:14 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 20:11 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: b65da7e2-9fd2-4a1e-8dd4-{{Gerrit|88ca65936ae4}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: cbbf114c-9a4c-4dd2-9b58-{{Gerrit|50da8bd896ca}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 5969beaf-72bf-4af9-ae4d-{{Gerrit|8e5c3331e78e}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:55 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: b8f7389d-79b4-411d-b3d5-{{Gerrit|ec8f92eba101}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:54 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:53 wm-bot2: Safe reboot of 'cloudvirt1051.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:53 wm-bot2: Unset cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:51 wm-bot2: Safe reboot of 'cloudvirt1049.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:51 wm-bot2: Unset cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:50 wm-bot2: Safe reboot of 'cloudvirt1048.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:50 wm-bot2: Unset cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:49 wm-bot2: Drained 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: c5dbfa7b-72fc-4156-8257-{{Gerrit|af224a725b78}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:36 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:33 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: c1a3e92c-cb04-46e4-ac5d-{{Gerrit|784c79601b05}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance (downtime id: 62fdc4ee-c8d5-4892-aeb9-{{Gerrit|a681cdcbc84b}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 9ec407cd-5c00-407a-a57d-{{Gerrit|794f6c68f947}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Set cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance (downtime id: 712683ea-d58f-484b-8efb-{{Gerrit|89d1c21aa0e3}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Safe rebooting 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Draining 'cloudvirt1049.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:29 wm-bot2: Safe rebooting 'cloudvirt1049.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Draining 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:28 wm-bot2: Safe rebooting 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:25 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
* 19:25 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
* 19:21 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:04 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 18fae8d8-7353-4f67-90d7-{{Gerrit|8df9b3fb1ccb}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:03 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:03 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:01 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 3d3cffa3-abc6-4901-83d9-{{Gerrit|3bcc4b02fd3c}}, use this to unset). - cookbook ran by andrew@bullseye
* 19:00 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 19:00 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:54 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:54 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:50 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:46 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
* 18:46 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye

=== 2022-10-15 ===
* 17:38 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki Slevinski # [[phab:T320867|T320867]]

=== 2022-10-13 ===
* 12:19 wm-bot2: OSDs (['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034'] ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 11:30 wm-bot2: OSDs (['cloudcephosd1025', 'cloudcephosd1026']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus

=== 2022-10-10 ===
* 14:01 dcaro: test2
* 01:22 andrewbogott: restarting designate-sink on cloudservices100[45], possible example of [[phab:T316614|T316614]]

=== 2022-10-09 ===
* 12:04 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki DatGuy # [[phab:T320301|T320301]]

=== 2022-10-07 ===
* 13:40 andrewbogott: dhinus is resetting rabbitmq cluster in an attempt to resolve a suspected (by Andrew) split-brain
* 11:33 arturo: rabbitmq-server.service @ cloudrabbit1002 is again up and running ([[phab:T320232|T320232]])
* 10:24 arturo: stopping rabbitmq-server.service @ cloudrabbit1002 ([[phab:T320232|T320232]])
* 10:19 arturo: restarting nova-conductor in all 3 cloudcontrols ([[phab:T320232|T320232]])
* 09:45 arturo: restarting rabbitmq-server.service @ cloudrabbit1002 ([[phab:T320232|T320232]])

=== 2022-10-06 ===
* 15:55 arturo: cloudnet1005 &amp; cloudnet1006 now in service. Secom cloudnet1003 &amp; cloudnet1004. Drop neutron agents, etc. ([[phab:T316284|T316284]])
* 11:54 arturo: rebooting cloudnet1005/1006 to see if they have the right network config ([[phab:T316284|T316284]])
* 11:50 arturo: set neutron l3 agents on cloudnet1005/1006 as down `root@cloudcontrol1005:~# neutron agent-update --admin-state-down &lt;uuid&gt;` ([[phab:T316284|T316284]])
* 11:40 arturo: [codfw1dev] rebooting both network nodes to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/839492
* 10:14 arturo: [codfw1dev] restart neutron-l3-agent on cloudnet2006-dev, it was dead

=== 2022-10-05 ===
* 14:40 wm-bot2: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T319418|T319418]]) - cookbook ran by fran@wmf3169
* 14:40 wm-bot2: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T319418|T319418]]) - cookbook ran by fran@wmf3169
* 14:28 arturo: adding cloudinstances2b-gw router to l3 agents on cloudnet1005/1006 ([[phab:T316284|T316284]])
* 13:11 wm-bot2: Added 1 new OSDs ['cloudcephosd1034.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 13:11 wm-bot2: Added OSD cloudcephosd1034.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 13:02 wm-bot2: Finished rebooting node cloudcephosd1034.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Rebooting node cloudcephosd1034.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Adding OSD cloudcephosd1034.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:58 wm-bot2: Adding new OSDs ['cloudcephosd1034.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169

=== 2022-10-04 ===
* 16:40 wm-bot2: Added 1 new OSDs ['cloudcephosd1033.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 16:40 wm-bot2: Added OSD cloudcephosd1033.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:34 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:30 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 14:20 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 14:17 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:59 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:56 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:55 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:55 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169

=== 2022-09-30 ===
* 14:52 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169
* 14:52 wm-bot2: Added OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:48 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 14:48 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 14:16 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 64eac5c6-4b1d-4269-98fd-{{Gerrit|8e5bed42ce40}}, use this to unset). ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T319025|T319025]]) - cookbook ran by andrew@buster
* 14:09 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:05 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: fb99c967-b974-4314-a2fa-{{Gerrit|31ed0e883dd3}}, use this to unset). - cookbook ran by andrew@buster
* 14:04 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 13:30 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169
* 13:30 wm-bot2: Added OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:26 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:26 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 13:16 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:16 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 13:08 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:08 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 12:45 wm-bot2: Finished rebooting node cloudcephosd1031.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:42 wm-bot2: Rebooting node cloudcephosd1031.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:41 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:41 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 12:26 arturo: [codfw1dev] cloudnet2005/2006-dev are now on a single NIC setup ([[phab:T318824|T318824]])
* 11:44 arturo: sysctl change (cleanup) on cloudnet1003/1004

=== 2022-09-27 ===
* 10:48 wm-bot2: Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:35 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:26 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:23 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:22 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:22 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:17 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:09 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169

=== 2022-09-26 ===
* 18:37 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 18:37 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:33 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:31 andrewbogott: rebooting cloudvirt1028 for [[phab:T317391|T317391]]
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance (downtime id: 6d1c4c53-76b9-493f-8c44-{{Gerrit|eb0413dfb9d0}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance (downtime id: 6bb80b65-616c-4e45-be4f-{{Gerrit|d2cdd8abb2bf}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance (downtime id: a3bba7e7-bcd3-482d-a486-{{Gerrit|06b6f53fd899}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:18 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8fb59651-fd42-4aee-a978-{{Gerrit|36bc7f79b4d5}}, use this to unset). - cookbook ran by andrew@buster
* 18:18 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:17 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:01 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:01 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:58 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:51 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:51 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:47 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:47 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:47 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:45 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:38 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: abfc35bb-331e-4d7e-bb92-{{Gerrit|35e675abce32}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:34 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:33 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: d4fe6ea8-76eb-45f7-b6cf-{{Gerrit|66f54ba9801c}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:33 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: c0671bda-9aae-4960-85be-{{Gerrit|58aaa9c8e021}}, use this to unset). - cookbook ran by andrew@buster
* 17:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:32 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:32 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:28 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:28 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:24 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: be6559f9-4d72-4492-b9b9-{{Gerrit|23c636d6554e}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:23 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:23 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:21 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 237dfb1b-5382-408d-aa35-{{Gerrit|1aa1cfe4c5af}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 088b97cd-0cf2-4c27-a67f-{{Gerrit|077ffa1c1c5e}}, use this to unset). - cookbook ran by andrew@buster
* 17:16 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:16 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 91625000-aa6d-4887-bf34-{{Gerrit|ef7785a4af4a}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:16 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:15 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:15 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:15 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:15 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 7364c4fe-9f6b-4539-b6fa-{{Gerrit|1e768b602a1b}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:14 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:14 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:12 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:12 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: cb8f8a21-73c0-4f54-9412-{{Gerrit|074d82582cb4}}, use this to unset). - cookbook ran by andrew@buster
* 17:11 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:11 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:09 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: d23dabd2-3bfc-4ce0-9533-{{Gerrit|cd1cf910f8e1}}, use this to unset). - cookbook ran by andrew@buster
* 17:08 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:08 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:05 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 4d7ddae1-f621-4dd5-8616-{{Gerrit|29b7699b692f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 4628dd6f-5b49-417f-b6ab-{{Gerrit|5c41992192b4}}, use this to unset). - cookbook ran by andrew@buster
* 17:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: c95529cf-b2a5-4553-bd6e-{{Gerrit|6ad4dcb2a105}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:01 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 36c68112-a964-4a7c-a093-{{Gerrit|8a6d481dea7d}}, use this to unset). - cookbook ran by andrew@buster
* 17:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:00 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:00 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:49 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:49 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:45 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:31 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 1244c159-65bb-476e-a702-{{Gerrit|8f43c253e499}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:30 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:30 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:22 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 13:19 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 13:18 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:18 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 12:32 dcaro: Changed the collation of labsdbaccount db to utf8mb4_bin ([[phab:T318047|T318047]])
* 10:14 arturo: deployed new version of maintain-dbusers ([[phab:T318047|T318047]])

=== 2022-09-25 ===
* 15:06 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:06 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: a050e47f-3a63-41ff-accb-{{Gerrit|3993c1e8b593}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:05 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:05 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:58 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:58 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 1a49d773-4dc9-4a6f-bd49-{{Gerrit|8663db77ce66}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:57 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:57 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:54 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: ab52dd42-68a6-4159-b345-{{Gerrit|e5625d209b57}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:53 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:53 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 10337415-3095-4834-8538-{{Gerrit|b3a64bd6b18d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:53 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:53 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 4cdbed3a-3775-4913-8c3b-{{Gerrit|afd57ae1ca55}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-24 ===
* 17:37 andrewbogott: restarting neutron api on cloudcontrol1006; cause of outage unknown
* 17:35 andrewbogott: restarting neutron-linuxbridge-agent on cloudvirt1022

=== 2022-09-22 ===
* 15:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:10 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04849437-a304-45a1-a037-{{Gerrit|538741a2f801}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:09 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:09 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:07 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:07 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:04 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 8810a919-61e7-4ba5-af7f-{{Gerrit|c41d1e1c8c8b}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:59 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:59 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:56 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: b6f8e0b6-f35c-41fd-8035-{{Gerrit|6efde61db3a3}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 784bff2c-8160-44dc-9bcf-{{Gerrit|ff23055a0527}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:40 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:40 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:39 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:38 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:38 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 21152e9e-bc67-4024-b68e-{{Gerrit|fd671c398642}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: f07ddb1d-e8da-43f3-8140-{{Gerrit|bc66789d37c8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:36 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:36 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:33 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:33 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:30 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 4e4b03fc-e3f1-440e-9097-{{Gerrit|cce5edaa1f08}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: d4bb2160-f492-4ddc-a5b7-{{Gerrit|eeee37007d14}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9bf00a6a-3697-4201-9a6e-{{Gerrit|76d499eccfa1}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:13 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:13 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:47 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f2888490-1804-4b23-b7b0-{{Gerrit|677853fb9ef7}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8a3e8b0a-ced3-4667-a121-{{Gerrit|66df673c7ed8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:43 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 35c761b3-f0de-4b23-9b11-{{Gerrit|2ed2e474c89f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 6121a0e1-754a-47fa-b51b-{{Gerrit|265f6386f396}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:40 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:40 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:39 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 189ae1bb-c69d-4eb0-aee9-{{Gerrit|90ab1f492aa8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:38 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:38 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:36 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: acc53776-68bf-47a4-bb82-{{Gerrit|c571b3df2945}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:31 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 07770ae7-3e7e-4615-8174-{{Gerrit|513767f95b1d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:30 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:30 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 7ae20a7d-8bff-4bc2-91c5-{{Gerrit|b0005c8164d1}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: b596f855-8019-49d2-9657-{{Gerrit|0d513ad7acb5}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:25 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 9172c785-d2d6-46e5-bed3-{{Gerrit|2f49d1033f78}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:23 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:23 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:19 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 85b4e0e2-e635-485f-9ce5-{{Gerrit|341981b3887f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 9494c283-02c8-42da-98c3-{{Gerrit|139b4d119ef8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: d181f301-249f-4b5a-bd85-{{Gerrit|47d87262d0ed}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-20 ===
* 21:02 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 21:02 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:58 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:58 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 1c4246a4-9cee-4423-8cd1-{{Gerrit|4f52f61d503f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:53 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:50 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:50 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:49 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 82665ecc-4431-48fe-b255-{{Gerrit|7e9d518be217}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:48 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:48 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:47 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: cd284562-e3b0-4504-9977-{{Gerrit|2b18032bbf3c}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:45 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 8fb2a646-16a5-4183-94a1-{{Gerrit|ee6bbbf029fa}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:44 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:44 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: d561fe45-1582-4cd8-bbc9-{{Gerrit|a356c39f3330}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 351c365c-0228-4907-a279-{{Gerrit|01795b1e62ce}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 3beec1dd-0132-4f5c-adce-{{Gerrit|5a7dc56060b6}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:30 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:29 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:28 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:28 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:26 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:24 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:20 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:16 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 353a8527-ad5e-4898-937c-{{Gerrit|303d16801e28}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: fc235146-f070-4723-9503-{{Gerrit|e20cbb877755}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: cdf712ac-c083-4a63-af83-{{Gerrit|514ca5cdc76d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:05 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:01 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:01 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 8f2d2d32-e99e-4e2c-9472-{{Gerrit|ca33b577d01f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 66ee7d87-cee1-4c8f-b5c3-{{Gerrit|5f58a6cea336}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:59 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:59 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:58 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:58 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:58 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 69035b38-9910-46f1-9582-{{Gerrit|3105c1ba4d16}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:57 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:57 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:52 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 5a94c9d1-c956-4b03-a92b-{{Gerrit|b112810906a4}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:51 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:51 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:50 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 216a9120-4b9a-4be3-99db-{{Gerrit|3fd5fdc25146}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:49 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:49 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 46fb1964-3cf4-47fd-ad79-{{Gerrit|475f33c8ed38}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: b04a4f8c-db82-4258-8608-{{Gerrit|2a78ec9193ed}}, use this to unset). - cookbook ran by andrew@buster
* 19:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 7ae7c258-ecb8-47d1-a991-{{Gerrit|3fa617e305bf}}, use this to unset). - cookbook ran by andrew@buster
* 19:45 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 377fb0e5-132e-4e7a-9079-{{Gerrit|37937c3d42bf}}, use this to unset). - cookbook ran by andrew@buster
* 19:44 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 6c4222ae-796e-4890-8820-{{Gerrit|9e7ce5438f2a}}, use this to unset). - cookbook ran by andrew@buster
* 19:43 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:43 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:38 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:34 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:34 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:31 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: eff98e2e-ae99-41d0-a68e-{{Gerrit|671d941fcf21}}, use this to unset). - cookbook ran by andrew@buster
* 19:30 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5310f1eb-0425-4f59-8fb5-{{Gerrit|1d9c6f6d2f23}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:29 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:25 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: d8ab4682-026a-4ddc-bfc5-{{Gerrit|48166bd9789a}}, use this to unset). - cookbook ran by andrew@buster
* 19:24 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:23 wm-bot2: Safe reboot of 'cloudvirt1048.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:23 wm-bot2: Unset cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: eb2f9d94-8ea5-48d5-a336-{{Gerrit|97dd407b1ce3}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Safe rebooting 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:18 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:14 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 305fe2d7-913e-4638-95c2-{{Gerrit|4e810dc6b2a3}}, use this to unset). - cookbook ran by andrew@buster
* 19:14 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5bb3a66b-9450-4230-8d09-{{Gerrit|d78473d72ba2}}, use this to unset). - cookbook ran by andrew@buster
* 19:13 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:13 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:07 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: b1e39b49-3a88-4e42-913c-{{Gerrit|9892eafad447}}, use this to unset). - cookbook ran by andrew@buster
* 19:06 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:05 andrewbogott: putting cloudvirt1049-1052 into 'ceph' pool, taking out of 'spare' pool. cloudvirt1053 will remain our only spare.
* 19:02 wm-bot2: Safe reboot of 'cloudvirt1049.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:02 wm-bot2: Unset cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:00 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:00 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:59 wm-bot2: Safe reboot of 'cloudvirt1051.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:59 wm-bot2: Unset cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Drained 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Set cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance (downtime id: 8f24ea75-e107-4a13-8bd6-{{Gerrit|950b941b8e03}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Draining 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Safe rebooting 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:57 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:57 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:56 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:56 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 5c3e8fa4-cd1e-43f4-a423-{{Gerrit|ba7fe2459d4e}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Drained 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Set cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance (downtime id: e3fe22d6-932c-4502-8838-{{Gerrit|82cdc4c8f1c6}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:54 wm-bot2: Draining 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:54 wm-bot2: Safe rebooting 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:53 wm-bot2: Safe reboot of 'cloudvirt1053.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:53 wm-bot2: Unset cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: fd824b92-b6ff-4631-a2d4-{{Gerrit|debb25d48223}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:51 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:49 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 52d5cd5b-0f56-453a-85f8-{{Gerrit|0b5e656ae7f9}}, use this to unset). - cookbook ran by andrew@buster
* 18:49 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:49 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: b92f66a3-a487-4515-b68b-{{Gerrit|1c257a57b893}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:48 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster

=== 2022-09-19 ===
* 20:07 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 4bfca6cd-dfbc-4a79-9203-{{Gerrit|b432bfeee5c2}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:41 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:21 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 70479325-609b-4349-9094-{{Gerrit|739eb9b3bb30}}, use this to unset). - cookbook ran by andrew@buster
* 19:21 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-09-14 ===
* 16:57 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
* 11:01 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:58 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:57 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:57 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:09 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:09 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:06 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:06 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 09:46 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-09-13 ===
* 12:16 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:16 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:11 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:11 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:36 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-09-10 ===
* 15:37 andrewbogott: restarting nova-conductor service (and possibly others, in response to lots of unanswered rabbitmq messages)

=== 2022-09-08 ===
* 19:12 andrewbogott: restarting nginx on proxy-03.project-proxy.eqiad1.wikimedia.cloud

=== 2022-09-07 ===
* 10:18 wm-bot2: Added OSD cloudcephosd1032.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:14 wm-bot2: Finished rebooting node cloudcephosd1032.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Rebooting node cloudcephosd1032.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Adding OSD cloudcephosd1032.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Adding new OSDs ['cloudcephosd1032.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:41 dhinus: Temporarily removing cloudcephosd1030 from Ceph cluster (https://phabricator.wikimedia.org/T314870)

=== 2022-08-30 ===
* 14:59 andrewbogott: manually marking most eqiad1 cloud* servers down in icinga for [[phab:T296561|T296561]]
* 10:43 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:39 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:38 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:38 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:39 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:14 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:11 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-25 ===
* 15:14 wm-bot2: Added 1 new OSDs ['cloudcephosd1029.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 15:14 wm-bot2: Added OSD cloudcephosd1029.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 15:02 wm-bot2: Finished rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:59 wm-bot2: Rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:58 wm-bot2: Adding OSD cloudcephosd1029.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:58 wm-bot2: Adding new OSDs ['cloudcephosd1029.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-24 ===
* 22:07 andrewbogott: replaced cloudservices1003 with cloudservices1005 [[phab:T304888|T304888]]
* 10:45 wm-bot2: Added 1 new OSDs ['cloudcephosd1028.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:45 wm-bot2: Added OSD cloudcephosd1028.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Finished rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:34 wm-bot2: Rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:33 wm-bot2: Adding OSD cloudcephosd1028.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:33 wm-bot2: Adding new OSDs ['cloudcephosd1028.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-23 ===
* 13:46 wm-bot2: Added 1 new OSDs ['cloudcephosd1027.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:46 wm-bot2: Added OSD cloudcephosd1027.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:27 wm-bot2: Finished rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:24 wm-bot2: Rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:22 wm-bot2: Adding OSD cloudcephosd1027.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:22 wm-bot2: Adding new OSDs ['cloudcephosd1027.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-21 ===
* 21:12 andrewbogott: restarted neutron-dhcp-agent on cloudnet1003. it was claiming to be unable to contact Rabbit but seems happy after a restart

=== 2022-08-20 ===
* 07:39 dcaro_away: cloudvirt1023 is back up, VMs are starting to recover ([[phab:T315718|T315718]])
* 07:23 dcaro_away: cloudvirt1023 seems to have gotten some hardware issue from racadm lclog view &quot;System CPU Resetting.&quot;, rebooting and doing memory checks ([[phab:T315718|T315718]])

=== 2022-08-19 ===
* 17:06 taavi: [codfw1dev] restart mariadb on clouddb2002-dev to pick up certificate config changes [[phab:T310795|T310795]]

=== 2022-08-18 ===
* 13:25 wm-bot2: Added 1 new OSDs ['cloudcephosd1026.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:25 wm-bot2: Added OSD cloudcephosd1026.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:15 wm-bot2: Finished rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:12 wm-bot2: Rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:10 wm-bot2: Adding OSD cloudcephosd1026.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:10 wm-bot2: Adding new OSDs ['cloudcephosd1026.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 07:29 dcaro: Starting up all the osd daemons on cloudcephosd1025 ([[phab:T314870|T314870]])

=== 2022-08-17 ===
* 10:50 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:49 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:50 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:49 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:49 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:16 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:16 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:16 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro

=== 2022-08-16 ===
* 22:39 andrewbogott: replacing the now-rebuilt cloudvirt1025 in 'ceph' aggregate and removing it from the 'maintenance' aggregate
* 17:41 andrewbogott: removing cloudvirt1025 from the 'ceph' aggregate and adding it to the 'maintenance' aggregate
* 17:40 andrewbogott: reimaging cloudvirt1025 after I accidentally deleted the hw raid
* 17:38 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1003@rbd#RBD --newhost cloudcontrol1005@rbd#RBD
* 17:37 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1004@rbd#RBD --newhost cloudcontrol1006@rbd#RBD
* 16:26 wm-bot2: Ceph cluster at eqiad1 set out of maintenance. - cookbook ran by dcaro@vulcanus
* 15:43 wm-bot2: Restarting the osd daemons from nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,c
* 15:42 wm-bot2: Finished restarting all the OSD daemons from the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
* 15:38 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:02 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 12:59 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus

=== 2022-08-14 ===
* 18:36 taavi: deleted the http keystone endpoints from the keystone service catalog

=== 2022-08-11 ===
* 13:57 andrewbogott: decommissioning cloudcontrol1003 + cloudcontrl1004. I backed up $home in case anyone needs their files.
* 08:42 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:42 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:42 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-10 ===
* 13:10 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station

=== 2022-08-04 ===
* 17:16 taavi: deleted all scheduler_fanout_ rabbit queues in an attempt to fix scheduling
* 16:32 taavi: restart neutron-l3-agent to pick up rabbit config changes
* 15:12 andrewbogott: stopping rabbitmq on cloudcontrol1xxx
* 09:57 taavi: stop wikitech_run_jobs.timer on labweb1001/1002, hosts pending decom

=== 2022-08-03 ===
* 20:55 andrewbogott: root@tools-checker-04:~# systemctl restart uwsgi-toolschecker_cron.service
* 20:41 andrewbogott: restarting  neutron-l3-agent.service on cloudnet1003 and 1004. The agent was routing properly but had lost touch with rabbitmq

=== 2022-08-02 ===
* 14:07 andrewbogott: shutting down codfw1dev ceph cluster according to https://docs.mirantis.com/mcp/q4-18/mcp-operations-guide/scheduled-maintenance-power-outage/power-off-ceph-cluster.html
* 13:54 andrewbogott: shutting down basically all of codfw1dev to support pdu maintenance -- all the ceph OSDs will lose power so best to have everything stopped.

=== 2022-07-27 ===
* 19:32 andrewbogott: switching the openstack.eqiad1.wikimedia.cloud endpoint from cloudcontrol1004 to 1006, https://gerrit.wikimedia.org/r/c/operations/dns/+/817878/2/templates/wikimediacloud.org#54
* 16:33 andrewbogott: here is a test message in the admin channel

=== 2022-07-25 ===
* 13:43 andrewbogott: pooling cloudweb100[34] and depooling labweb100[12] for testing in prep for decomming labweb100[12]

=== 2022-07-22 ===
* 16:41 taavi: depool cloudweb1003/1004 since horizon seems to be having issues
* 16:22 taavi: pooling cloudweb1003/1004 now that grant issues are sorted

=== 2022-07-21 ===
* 18:26 andrewbogott: depooling cloudweb1003 and 1004 for wikitech, horizon, striker -- pending db grant changes
* 18:06 andrewbogott: pooling cloudweb1003 and 1004 for wikitech, horizon, striker

=== 2022-07-20 ===
* 18:02 dcaro: things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 ([[phab:T313400|T313400]])
* 17:45 bd808: `sudo service striker restart` on labweb1002
* 17:43 bd808: `sudo service striker restart` on labweb1001
* 17:10 dcaro: things seem stable, trying to bring up a fourth rabbit node, cloudcontrol1006 ([[phab:T313400|T313400]])
* 16:26 dcaro: things seem stable, trying to bring up a third, cloudcontrol1005 ([[phab:T313400|T313400]])
* 15:51 dcaro: things seem stable now with one rabbit node, trying to bring up a second ([[phab:T313400|T313400]])
* 14:16 dcaro: stopping rabbin on cloudcontrol1004, leaving only 1003 alive ([[phab:T313400|T313400]])
* 13:17 dcaro: restarting the whole rabbit cluster ([[phab:T313400|T313400]])

=== 2022-07-19 ===
* 16:30 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:30 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 16:26 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:18 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:18 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 16:14 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:01 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:01 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:57 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:57 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 3da2d4f6-5c5b-4a21-9a0b-{{Gerrit|2b010960ed6a}}, use this to unset). - cookbook ran by andrew@buster
* 15:56 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:49 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 8a10505a-107d-4e78-ba84-{{Gerrit|363a7dea8d69}}, use this to unset). - cookbook ran by andrew@buster
* 15:47 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 752aa58b-44d4-4340-b05d-{{Gerrit|911b14f2314d}}, use this to unset). - cookbook ran by andrew@buster
* 15:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: f89f0851-ce3f-4a92-899e-{{Gerrit|0d4638acc9c3}}, use this to unset). - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:45 andrewbogott: adding new hosts to the 'ceph' aggregate: cloudvirt1046, 1047, 1048
* 15:44 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:44 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:42 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:42 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:40 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:38 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:37 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:37 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:33 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:32 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 32d9fdb7-6c42-4cf4-9950-{{Gerrit|6e2442255161}}, use this to unset). - cookbook ran by andrew@buster
* 15:31 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:31 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:16 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 56ce1388-792c-442c-bb89-{{Gerrit|e4c3869b0acf}}, use this to unset). - cookbook ran by andrew@buster
* 15:15 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:15 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:14 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 3f84bf2e-24de-4ebe-8117-{{Gerrit|0f4a5de29d09}}, use this to unset). - cookbook ran by andrew@buster
* 15:14 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:14 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:13 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:13 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:12 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 5418e923-997c-49ae-b937-{{Gerrit|35fc0c3038eb}}, use this to unset). - cookbook ran by andrew@buster
* 15:12 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: c3dfdc2b-0725-4d01-87f2-{{Gerrit|440a5ac4694c}}, use this to unset). - cookbook ran by andrew@buster
* 15:11 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:09 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:04 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:04 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:00 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:54 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:54 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:50 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:46 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 49dbf7de-c58a-4b68-bc7b-{{Gerrit|f001a047f4c7}}, use this to unset). - cookbook ran by andrew@buster
* 14:46 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:46 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:44 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: a4e95168-39b5-452d-8fce-{{Gerrit|7875ae44a62b}}, use this to unset). - cookbook ran by andrew@buster
* 14:44 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:44 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:43 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: b46234ee-9900-4708-a815-{{Gerrit|4324379be48c}}, use this to unset). - cookbook ran by andrew@buster
* 14:43 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:43 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:42 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:39 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:39 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:38 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:38 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 12bc87a9-7602-4795-818c-{{Gerrit|f7151b1c9ead}}, use this to unset). - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:35 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: ba787a26-62db-4662-ade2-{{Gerrit|2e3061b7390d}}, use this to unset). - cookbook ran by andrew@buster
* 14:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:29 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 6f0e79c6-2d91-478e-92df-{{Gerrit|0f65de48212f}}, use this to unset). - cookbook ran by andrew@buster
* 14:29 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 7f577c0f-0182-498f-b198-{{Gerrit|307d51df8c3a}}, use this to unset). - cookbook ran by andrew@buster
* 14:28 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: bd1bff92-bfb8-483d-a719-{{Gerrit|13fbdf3d8b21}}, use this to unset). - cookbook ran by andrew@buster
* 14:27 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:27 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: c9cd7dbb-c5ad-4364-a8e0-{{Gerrit|afb52ef98683}}, use this to unset). - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 4deca8a4-f73f-4b42-b4be-{{Gerrit|acfe61421ed0}}, use this to unset). - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 6d0d58b2-8450-480d-94f2-{{Gerrit|93a257f25575}}, use this to unset). - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:13 dcaro: deleting all the leftover fullstack images (was due to max number of mysql connections reached)
* 04:51 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:47 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:46 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 432747a2-9a15-44d4-ba8a-{{Gerrit|4e7e87eb8b76}}, use this to unset). - cookbook ran by andrew@buster
* 04:45 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1c5adb8c-efcf-4036-bf22-{{Gerrit|0eef364ae399}}, use this to unset). - cookbook ran by andrew@buster
* 04:44 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:44 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:41 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 26feb897-9eb0-40b6-9525-{{Gerrit|4cf44f81638d}}, use this to unset). - cookbook ran by andrew@buster
* 04:41 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:40 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:40 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:39 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: cd2424d3-c9c9-4837-93b1-{{Gerrit|a6d8271e7873}}, use this to unset). - cookbook ran by andrew@buster
* 04:39 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:39 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1e85ec01-81c4-4664-b14f-{{Gerrit|b1cf6417988c}}, use this to unset). - cookbook ran by andrew@buster
* 04:38 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f498aaa4-fad3-42f8-92f4-{{Gerrit|48a98c108456}}, use this to unset). - cookbook ran by andrew@buster
* 04:37 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 0ed281f3-d9b2-4c0a-bf88-{{Gerrit|db42fe848b2f}}, use this to unset). - cookbook ran by andrew@buster
* 04:35 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f417dffc-fa1b-4e65-938a-{{Gerrit|49e54b6fac6d}}, use this to unset). - cookbook ran by andrew@buster
* 04:34 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:33 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 923926e0-7194-4e13-98c4-{{Gerrit|a4f703b2907b}}, use this to unset). - cookbook ran by andrew@buster
* 04:32 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:32 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:29 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: ca8f729b-92ad-45ec-b5da-{{Gerrit|55987b0fc9c2}}, use this to unset). - cookbook ran by andrew@buster
* 04:29 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:29 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: abe6992d-dfba-4e50-993d-{{Gerrit|de0a551a177a}}, use this to unset). - cookbook ran by andrew@buster
* 04:24 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:24 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: df91107d-bf34-49db-9756-{{Gerrit|44ad06162683}}, use this to unset). - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f45ec0b1-83c9-49e0-a3a8-{{Gerrit|897d5a48414f}}, use this to unset). - cookbook ran by andrew@buster
* 04:23 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:18 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 504ae503-bfad-41bd-9079-{{Gerrit|fb53a9c82a62}}, use this to unset). - cookbook ran by andrew@buster
* 04:17 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:17 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 8410faf9-667d-443d-bd17-{{Gerrit|bb3ca8e7f725}}, use this to unset). - cookbook ran by andrew@buster
* 04:15 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:11 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 40441a37-0dd7-44a8-960b-{{Gerrit|f74f98f53618}}, use this to unset). - cookbook ran by andrew@buster
* 04:10 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:10 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:08 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 517cff2b-a643-4714-a31f-{{Gerrit|6bbe0767656a}}, use this to unset). - cookbook ran by andrew@buster
* 04:08 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 50b63ad6-4d8b-4fcf-9c17-{{Gerrit|27a106b6ec52}}, use this to unset). - cookbook ran by andrew@buster
* 04:07 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: d912bdf5-54c7-490a-bdbe-{{Gerrit|ba9a23f07f4b}}, use this to unset). - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: e75f16d0-e42f-4d72-a3bd-{{Gerrit|be0cbc5f200a}}, use this to unset). - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 87f6c417-3703-4f73-9876-{{Gerrit|20f6767dc8e1}}, use this to unset). - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: be14f869-fdb0-4c38-84ff-{{Gerrit|3d1e00d227eb}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: feb93ea7-2c64-4831-b140-{{Gerrit|dad8311dbcef}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: c65a23e3-a809-4a75-88e4-{{Gerrit|29a8b351ecdb}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:00 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:00 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:57 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:56 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8c8f64ca-a4e8-47cf-a2ce-{{Gerrit|23059109fb6a}}, use this to unset). - cookbook ran by andrew@buster
* 03:55 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:55 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:54 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 9d0457e8-a861-46ce-ac35-{{Gerrit|53975a46018e}}, use this to unset). - cookbook ran by andrew@buster
* 03:53 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:53 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:40 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:38 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:30 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 851d3b34-68f7-4c57-920b-{{Gerrit|2a64a9ea573d}}, use this to unset). - cookbook ran by andrew@buster
* 03:29 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:29 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:17 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 1060ab82-d53a-4d48-8e15-{{Gerrit|3198cabcfc2f}}, use this to unset). - cookbook ran by andrew@buster
* 03:17 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:17 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: dde1dc09-44b5-42af-ac49-{{Gerrit|c58dbbae7cb4}}, use this to unset). - cookbook ran by andrew@buster
* 03:14 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: b59fa99b-8713-4e63-8c56-{{Gerrit|08bd71fdfbe3}}, use this to unset). - cookbook ran by andrew@buster
* 03:13 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:07 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 3547c843-6b15-4151-b452-{{Gerrit|6f618a886b7b}}, use this to unset). - cookbook ran by andrew@buster
* 03:07 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 6f5808d0-afb2-4222-ae8e-{{Gerrit|4d953d49cd63}}, use this to unset). - cookbook ran by andrew@buster
* 03:06 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: da738ba3-883f-4d76-bb33-{{Gerrit|7eb551de5fc2}}, use this to unset). - cookbook ran by andrew@buster
* 03:05 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:05 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:04 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:04 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:03 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f90ca4c1-78af-46bd-9262-{{Gerrit|0fa4edc2898e}}, use this to unset). - cookbook ran by andrew@buster
* 03:02 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:00 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:00 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 62383663-9ae6-4a15-a2a6-{{Gerrit|62be06adbc96}}, use this to unset). - cookbook ran by andrew@buster
* 02:59 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:59 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:59 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 6e0da344-49ad-451f-8b8d-{{Gerrit|ea8f853a6f89}}, use this to unset). - cookbook ran by andrew@buster
* 02:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:52 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: c9ec5b7c-aa9f-4afd-a417-{{Gerrit|54c52c07ca47}}, use this to unset). - cookbook ran by andrew@buster
* 02:51 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:51 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:48 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 5509c648-58f6-49fd-90e8-{{Gerrit|ac9aa66e8b04}}, use this to unset). - cookbook ran by andrew@buster
* 02:48 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:48 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: af4ba9ad-e765-4087-86aa-{{Gerrit|9111c0821175}}, use this to unset). - cookbook ran by andrew@buster
* 02:46 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:45 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:45 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: 6da98123-8838-4d2e-8659-{{Gerrit|d769bfa292fe}}, use this to unset). - cookbook ran by andrew@buster
* 02:44 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:44 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:43 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 02:43 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:42 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:42 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 91860079-8fff-40db-9de7-{{Gerrit|17741d528ad6}}, use this to unset). - cookbook ran by andrew@buster
* 02:41 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:41 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:39 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: a6a0f414-d305-4a09-87d2-{{Gerrit|066eec83642b}}, use this to unset). - cookbook ran by andrew@buster
* 02:39 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 6000898c-4e5e-49c8-8841-{{Gerrit|3262d6da0366}}, use this to unset). - cookbook ran by andrew@buster
* 02:37 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:37 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 24a8b50e-5c04-40e6-a732-{{Gerrit|f0232b83e240}}, use this to unset). - cookbook ran by andrew@buster
* 02:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8d259aca-1803-4346-b5a4-{{Gerrit|a447673b3537}}, use this to unset). - cookbook ran by andrew@buster
* 02:34 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:33 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 704aca7f-0069-4a4a-b8de-{{Gerrit|1aba7b9ca7ef}}, use this to unset). - cookbook ran by andrew@buster
* 02:32 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04c0fba3-e2cb-40b4-abfd-{{Gerrit|d24165c9bd55}}, use this to unset). - cookbook ran by andrew@buster
* 02:32 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 25e5e0c8-5c27-460e-9b44-{{Gerrit|1cbd1c2fb551}}, use this to unset). - cookbook ran by andrew@buster
* 02:28 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:25 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5e62968a-0bb6-43b1-ae5e-{{Gerrit|e352442ef976}}, use this to unset). - cookbook ran by andrew@buster
* 02:24 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:24 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:21 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5b1a8e8f-8699-433d-b764-{{Gerrit|197862dc5cf1}}, use this to unset). - cookbook ran by andrew@buster
* 02:20 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:20 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:15 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:15 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:13 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 299c466e-602b-4a8e-bb37-{{Gerrit|156553dbe426}}, use this to unset). - cookbook ran by andrew@buster
* 02:13 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:13 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:12 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:12 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:03 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6a64d619-c484-411f-ad9d-{{Gerrit|89d43b063737}}, use this to unset). - cookbook ran by andrew@buster
* 02:02 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:02 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 301309a2-a8b5-4698-98d6-{{Gerrit|bb4aa0a75e45}}, use this to unset). - cookbook ran by andrew@buster
* 00:46 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: b2f8c6a2-7224-4fb1-8e12-{{Gerrit|dff6dda02380}}, use this to unset). - cookbook ran by andrew@buster
* 00:46 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:40 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:40 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:38 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: c0f62e0e-7865-4e77-a238-{{Gerrit|d707ceed53a8}}, use this to unset). - cookbook ran by andrew@buster
* 00:38 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:38 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:36 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 96a368d5-232f-4187-b0bb-{{Gerrit|5923a50fff71}}, use this to unset). - cookbook ran by andrew@buster
* 00:36 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:36 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:35 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 31b6bfcb-81cb-4d89-b977-{{Gerrit|190899ea2e64}}, use this to unset). - cookbook ran by andrew@buster
* 00:35 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:35 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:28 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 50f763d4-36ea-4241-8e71-{{Gerrit|cc6aa20ccc9e}}, use this to unset). - cookbook ran by andrew@buster
* 00:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: cd3d17af-ee15-4a9f-8750-{{Gerrit|66b97a46ea12}}, use this to unset). - cookbook ran by andrew@buster
* 00:27 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 9b075166-d424-42c0-8e06-{{Gerrit|f74a47abb798}}, use this to unset). - cookbook ran by andrew@buster
* 00:21 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-07-18 ===
* 23:11 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: e0aab64f-d911-4d7e-9a97-{{Gerrit|c59d9868ceea}}, use this to unset). - cookbook ran by andrew@buster
* 23:11 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 447d9c7a-a00e-41db-93ed-{{Gerrit|6d2bfc66e5df}}, use this to unset). - cookbook ran by andrew@buster
* 23:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:01 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 46c2ddf2-50c3-4824-ba72-{{Gerrit|d7fb3fa4c5e0}}, use this to unset). - cookbook ran by andrew@buster
* 23:01 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6facdf4f-7bb8-42cd-8ac8-{{Gerrit|b4412fc4cf70}}, use this to unset). - cookbook ran by andrew@buster
* 23:00 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:52 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 23190573-fc0d-4872-8343-{{Gerrit|e35b84132028}}, use this to unset). - cookbook ran by andrew@buster
* 22:51 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:51 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:51 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 990313c4-5fa6-4de7-a8bf-{{Gerrit|b361f0c79e90}}, use this to unset). - cookbook ran by andrew@buster
* 22:50 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:50 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:48 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:48 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:42 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 59367d39-bd2f-47fd-becd-{{Gerrit|255bf823ff12}}, use this to unset). - cookbook ran by andrew@buster
* 22:42 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:42 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:40 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 38602815-d187-4455-9676-{{Gerrit|59607505bba7}}, use this to unset). - cookbook ran by andrew@buster
* 22:39 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 00f21e1b-d014-4bc5-995f-{{Gerrit|e25201fc77c4}}, use this to unset). - cookbook ran by andrew@buster
* 22:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:31 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: d05d07b2-8c60-46de-a89e-{{Gerrit|76987901901b}}, use this to unset). - cookbook ran by andrew@buster
* 22:31 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 47f6b9e6-ceab-4fb6-8f76-{{Gerrit|0de4341d0841}}, use this to unset). - cookbook ran by andrew@buster
* 22:31 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:31 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 599042a5-da25-4985-b502-{{Gerrit|328fe40a72d5}}, use this to unset). - cookbook ran by andrew@buster
* 22:29 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:29 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:29 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: a554d107-c14d-41f8-b180-{{Gerrit|093e2cf73e72}}, use this to unset). - cookbook ran by andrew@buster
* 22:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9fef70c7-6f7d-4b33-a9bc-{{Gerrit|8c39630dc182}}, use this to unset). - cookbook ran by andrew@buster
* 22:26 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:02 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 3744c0fa-084d-4198-98ca-{{Gerrit|66c7a5210f41}}, use this to unset). - cookbook ran by andrew@buster
* 22:02 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 01caf1e1-df64-474a-94d5-{{Gerrit|f37751e23d62}}, use this to unset). - cookbook ran by andrew@buster
* 22:01 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:59 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 3018c2fc-9cd5-45c8-a160-{{Gerrit|d5ad5728c8d5}}, use this to unset). - cookbook ran by andrew@buster
* 21:59 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:59 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: d760f916-32ea-4194-8974-{{Gerrit|f36064a9866c}}, use this to unset). - cookbook ran by andrew@buster
* 21:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:00 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: cb2a38f3-e9ca-4bf6-989c-{{Gerrit|a27be2b2d88c}}, use this to unset). - cookbook ran by andrew@buster
* 20:59 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:58 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 93fa9477-54b8-46e7-8508-{{Gerrit|a2d492528d1f}}, use this to unset). - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f6cf5b29-ce8b-426e-af1c-{{Gerrit|cab74e0c4a3a}}, use this to unset). - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: fe26e612-d76e-4120-b60a-{{Gerrit|4f300add224f}}, use this to unset). - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 2abd94be-b6d8-4042-a9ce-{{Gerrit|998e41d932f4}}, use this to unset). - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 10973c71-30d8-44bf-a310-{{Gerrit|d0c1af76d399}}, use this to unset). - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f8f1b214-ec76-4157-a8e6-{{Gerrit|6374e5d14608}}, use this to unset). - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:11 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 95dae613-2b04-472e-a8f9-{{Gerrit|e71c3fbb8d0e}}, use this to unset). - cookbook ran by andrew@buster
* 20:10 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: c804f4db-ea63-44e9-aff8-{{Gerrit|fea54c238345}}, use this to unset). - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 597a901d-4624-43de-b986-{{Gerrit|c4c6c2fcf80c}}, use this to unset). - cookbook ran by andrew@buster
* 20:03 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:03 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: fdceed0d-6fa0-4806-8fb6-{{Gerrit|3b321a5a1623}}, use this to unset). - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:37 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: b0904345-9aa3-4202-b992-{{Gerrit|78644141517c}}, use this to unset). - cookbook ran by andrew@buster
* 19:36 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:31 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:31 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-07-13 ===
* 14:48 bd808: Added Tavvi as member of #acl*wmcs-team

=== 2022-07-12 ===
* 12:04 wm-bot2: Ceph cluster at &lt;nowiki&gt;{&lt;/nowiki&gt;self.deployment&lt;nowiki&gt;}&lt;/nowiki&gt; set out of maintenance. - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Set the ceph cluster for codfw1dev in maintenance, alert silence ids: db32805f-a033-4e0e-8fc7-{{Gerrit|ed0c2e9f8be1}},f4e698f0-4b51-4b07-9ba8-{{Gerrit|0b296ca1c4fb}},39ad5325-44ed-44d1-bba5-{{Gerrit|91a85c7a8401}},a584a3c6-9dae-41e9-ac82-{{Gerrit|a91cd35af8fb}} - cookbook ran by dcaro@vulcanus
* 08:37 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2005-dev', 'cloudnet2006-dev'] - cookbook ran by dcaro@vulcanus
* 08:37 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:31 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2005-dev,cloudnet2006-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-08 ===
* 15:57 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-07-07 ===
* 07:17 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T312509|T312509]]) - cookbook ran by dcaro@vulcanus
* 07:12 wm-bot2: Rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T312509|T312509]]) - cookbook ran by dcaro@vulcanus

=== 2022-07-06 ===
* 17:50 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: ['8a5b9eee-48c0-474d-8277-faeb05a2ea61', '65aad0fc-d887-47a3-b20c-d1ed461a2411', '86b078ae-3a27-4063-8c7c-198a2fe0c172'] - cookbook ran by dcaro@vulcanus

=== 2022-07-04 ===
* 13:27 wm-bot2: Rebooting cloudgw host cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot2: Rebooted cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Rebooting cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Rebooting all the cloudgw nodes from the eqiad1 deployment: cloudgw1001.eqiad.wmnet,cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:13 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 13:12 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:09 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:56 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:52 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:56 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 10:56 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:51 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:51 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:47 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:47 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:41 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 09:07 wm-bot2: Rebooting cloudnet host cloudnet1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:06 wm-bot2: Rebooted cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot2: Rebooting cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot2: Rebooting all the cloudnet nodes cloudnet1003,cloudnet1004 - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:53 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:53 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:47 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:47 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:55 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:44 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 07:44 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:36 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:36 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:09 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:05 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:04 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-03 ===
* 21:27 andrewbogott: rebuilding rabbit cluster in codfw1dev to get rid of some queues so unresponsive that they can't otherwise be deleted

=== 2022-07-02 ===
* 11:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-07-01 ===
* 15:35 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:30 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:29 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:25 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:42 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 14:42 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:32 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:02 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:58 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:58 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:54 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:52 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:37 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:34 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:30 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

=== 2022-06-30 ===
* 18:17 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 18:13 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:45 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 06:52 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-06-29 ===
* 08:45 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 08:41 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

=== 2022-06-28 ===
* 13:03 taavi: grant the tools project access to the g3.cores16.ram64.disk20.10xiops flavor [[phab:T301949|T301949]]

=== 2022-06-22 ===
* 16:48 taavi: restart designate-*.service on both cloudservices nodes
* 16:44 taavi: restart nova-conductor on all the cloudcontrol nodes
* 12:50 andrewbogott: rebooting each eqiad1 cloudcontrol node in hopes of getting a baseline re: openstack instability

=== 2022-06-21 ===
* 04:48 andrewbogott: stopping nova-fullstack agent on cloudcontrol1003; it's going to page us otherwise and we're all AFK tomorrow
* 04:02 andrewbogott: restarting rabbitmq on cloudcontrol100x (one at a time)

=== 2022-06-17 ===
* 17:53 andrewbogott: switching to a new python-based health check for galera and haproxy. This may make things more stable, or it may not. [[phab:T310664|T310664]]
* 06:15 taavi: restart neutron-linuxbridge-agent on cloudvirt1046

=== 2022-06-15 ===
* 11:34 taavi: restart neutron-linuxbridge-agent on cloudvirt1022

=== 2022-06-14 ===
* 16:26 wm-bot2: OSDs (['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloudcephosd1021', 'cl
* 14:38 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudceph
* 12:55 wm-bot2: OSDs (['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 12:44 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus

=== 2022-06-13 ===
* 11:14 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:07 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:05 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:04 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:03 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 09:15 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus

=== 2022-06-06 ===
* 13:21 andrewbogott: restarting mysql/galera on cloudcontrol100x in an attempt to stabilize some flapping there

=== 2022-06-02 ===
* 07:51 taavi: restart neutron-linuxbridge-agent.service on cloudvirt1034 [[phab:T309732|T309732]]
* 00:37 andrewbogott: updated nameservers for codfw1dev instances via 'openstack subnet set --dns-nameserver etc.'

=== 2022-06-01 ===
* 17:11 andrewbogott: restarting designate services in cloudservices1xxx hosts
* 16:37 taavi: root@cloudcontrol1005:~# cinder reset-state --state available 491066a4-16a9-4ce4-a9f6-{{Gerrit|182660616d77}} for [[phab:T309659|T309659]]

=== 2022-05-30 ===
* 17:43 andrewbogott: restarting neutron-rpc and neutron-api services on cloudcontrol1xxx

=== 2022-05-29 ===
* 14:55 andrewbogott: restarting nova services on all eqiad1 cloudcontrol nodes to recover from rabbit breakage
* 14:15 andrewbogott: restarting rabbitmq on all eqiad1 cloudcontrol nodes (one at a time)

=== 2022-05-25 ===
* 20:01 balloons: clean up cinder backup a bit, restart service due to network outage
* 20:01 balloons: cloudvirt1029 restarted nova due to network outage

=== 2022-05-19 ===
* 15:21 andrewbogott: resetting password for the 'troveguest' rabbitmq user. I think I may have broken this during a recent rebuild of the rabbitmq cluster

=== 2022-05-18 ===
* 15:42 andrewbogott: updated the 'debian-11.0-bullseye' glance image with a fresh build

=== 2022-05-14 ===
* 11:33 taavi: deleted projects 'ores' and 'ores-staging' [[phab:T308102|T308102]]

=== 2022-05-13 ===
* 06:20 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 06:20 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:16 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:16 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:15 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:15 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:11 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 06:11 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:10 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:10 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:09 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:07 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:06 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:06 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:05 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:51 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:50 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:50 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:47 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:47 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:44 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:44 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:41 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:40 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:40 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:38 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:37 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:37 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:30 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:29 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:29 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:19 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:12 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:12 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:08 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:01 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:52 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:48 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:48 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:44 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:31 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:24 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:21 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:21 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:18 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:37 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:36 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:36 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:34 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:34 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:27 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:27 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:25 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:25 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:55 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:55 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:04 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:23 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 01:23 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:19 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:01 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:01 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:25 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 00:25 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:20 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:19 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:19 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:11 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 00:11 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:07 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-12 ===
* 23:55 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:55 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:23 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:23 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:20 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:17 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:17 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:13 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:57 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:56 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:56 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:55 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:55 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:54 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:54 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:53 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:52 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:52 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:51 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:44 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:44 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:42 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:36 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:36 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:33 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:33 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:30 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:22 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:22 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:21 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:21 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:11 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:11 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:08 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:08 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:07 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:07 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:05 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:04 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:04 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:00 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:55 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:55 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:53 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:50 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:48 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:48 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:44 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:44 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:43 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:43 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:31 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:23 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:22 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:22 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:11 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:07 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:00 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:58 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:35 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:35 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:06 andrewbogott: stopping nfs-server on labstore1004 in preparation for reboot
* 04:12 andrewbogott: rebooting primary bastion (bastion-eqiad1-03.bastion.eqiad1.wikimedia.cloud) in hopes of resolving a problem with ssh proxying

=== 2022-05-11 ===
* 18:48 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:39 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:38 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:38 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:04 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:03 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:03 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 08:56 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 08:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 07:53 dcaro: test
* 04:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:43 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:43 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:39 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:23 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:09 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:08 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:08 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:04 andrewbogott: reset and recreated the rabbitmq cluster in eqiad1 to get around some broken queues.
* 03:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:25 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:25 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-10 ===
* 21:43 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:40 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:40 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:35 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:32 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:05 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:02 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:55 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:55 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:47 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:46 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:45 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:45 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:40 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:39 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:39 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:37 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:36 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:33 wm-bot: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 19:33 wm-bot: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:29 wm-bot: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:06 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:05 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:05 wm-bot: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:41 andrewbogott: rebooting cloud*-dev for [[phab:T307668|T307668]]
* 13:59 taavi: manually attached [[User:Dreamy Jazz]] to wikitech for a password reset (https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Manually_associate_an_LDAP_account_with_wikitech)

=== 2022-05-07 ===
* 01:33 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:32 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:30 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:21 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:18 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster

=== 2022-05-03 ===
* 20:38 andrewbogott: upgrading clouddb2001-dev in place
* 18:18 taavi: updated 'puppet-enc' endpoints on the keystone catalog to use https and port 443

=== 2022-05-02 ===
* 16:56 dcaro: rebooting cloudmetrics1001

=== 2022-04-29 ===
* 14:22 andrewbogott: changing login.toolforge.org, bastion.toolforge.org, and dev.toolforge.org dns entries to refer to the new Buster bastions [[phab:T277653|T277653]] https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Timeline

=== 2022-04-27 ===
* 14:51 wm-bot: Finished rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloud
* 14:50 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:41 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:41 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot: Finished rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot: Rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot: Finished rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:01 wm-bot: Rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:01 wm-bot: Finished rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Finished rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:44 wm-bot: Rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:43 wm-bot: Finished rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:39 wm-bot: Rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:39 wm-bot: Finished rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:35 wm-bot: Rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:35 wm-bot: Finished rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Finished rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:26 wm-bot: Rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:26 wm-bot: Finished rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot: Rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot: Finished rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot: Rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot: Finished rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot: Rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot: Finished rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Finished rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:57 wm-bot: Rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:57 wm-bot: Finished rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Finished rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Finished rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:46 wm-bot: Rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:46 wm-bot: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024 - cookbo
* 12:15 wm-bot: Finished rebooting the nodes ['cloudcephmon1001', 'cloudcephmon1002', 'cloudcephmon1003'] - cookbook ran by dcaro@vulcanus
* 12:15 wm-bot: Finished rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:12 wm-bot: Rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:12 wm-bot: Finished rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:09 wm-bot: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:09 wm-bot: Finished rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:07 wm-bot: Rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:07 wm-bot: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:02 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:02 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:59 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:59 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 11:55 wm-bot: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] - cookbook ran by dcaro@vulcanus
* 11:55 wm-bot: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot: Rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:47 wm-bot: Rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:47 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:43 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:43 wm-bot: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev - cookbook ran by dcaro@vulcanus

=== 2022-04-26 ===
* 10:36 taavi: [codfw1dev] updated designate pool to 2004/2005-dev according to the instructions on https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS/Designate#Initial_designate/pdns_node_setup

=== 2022-04-22 ===
* 10:33 taavi: [codfw1dev] restart designate-sink on both new cloudservices host to fix rabbitmq connectivity

=== 2022-04-21 ===
* 05:38 andrewbogott: replaced cloudservices200[2,3] with cloudservices200[4,5]

=== 2022-04-19 ===
* 15:29 andrewbogott: stopping all VMs on cloudvirt1019, reimaging host

=== 2022-04-18 ===
* 15:23 andrewbogott: reimaging cloudvirt1020, leaving VMs in place
* 13:40 andrewbogott: shutting down many codfdfw1dev servers (including network infra!) for [[phab:T305469|T305469]]

=== 2022-04-14 ===
* 20:14 andrewbogott: restarting nova-api and nova-conductor services in a superstitious attempt to reduce open DB connections

=== 2022-04-13 ===
* 22:01 andrewbogott: restarting galera on cloudcontrols (one by one) to clear open connections

=== 2022-04-11 ===
* 15:59 taavi: created cloudinfra.wmcloud.org zone

=== 2022-04-09 ===
* 19:55 andrewbogott: reimaging cloudbackup1001-dev to bullseye
* 19:37 taavi: add 'puppet-enc' service &amp; endpoint to keystone [[phab:T274666|T274666]]
* 19:25 andrewbogott: reimaging cloudbackup1002-dev to bullseye

=== 2022-04-07 ===
* 12:51 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T305631|T305631]]) - cookbook ran by arturo@nostromo

=== 2022-04-06 ===
* 09:12 arturo: [codf1dev] installing python3-eventlet 0.30.2-5~bpo11+1 on all required servers (cloudvirt, cloudnet, cloudcontrol) ([[phab:T305157|T305157]])
* 08:45 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudvirt2003-dev ([[phab:T305157|T305157]])
* 08:42 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudcontrol servers ([[phab:T305157|T305157]])
* 08:24 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudvirt2003-dev ([[phab:T305157|T305157]])
* 08:20 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudcontrol servers ([[phab:T305157|T305157]])

=== 2022-03-30 ===
* 11:20 arturo: apply urpf strict filter to eqiad cloud-hosts vlan - [[phab:T285461|T285461]]

=== 2022-03-29 ===
* 10:02 dcaro: restarting keystone ([[phab:T304918|T304918]])

=== 2022-03-23 ===
* 22:53 wm-bot: Drained 'cloudvirt1045.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:38 wm-bot: Drained 'cloudvirt1044.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:12 wm-bot: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:12 wm-bot: Draining 'cloudvirt1045.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:08 wm-bot: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:07 wm-bot: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:06 wm-bot: Draining 'cloudvirt1044.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:06 wm-bot: Draining 'cloudvirt1043.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:54 wm-bot: Drained 'cloudvirt1042.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:19 wm-bot: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:19 wm-bot: Draining 'cloudvirt1042.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:12 wm-bot: Drained 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:12 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:09 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:07 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:04 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:55 wm-bot: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:54 wm-bot: Draining 'cloudvirt1041.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:30 wm-bot: Drained 'cloudvirt1039.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:14 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:14 wm-bot: Draining 'cloudvirt1039.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:44 wm-bot: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:43 wm-bot: Draining 'cloudvirt1038.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:19 wm-bot: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:18 wm-bot: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:13 wm-bot: Drained 'cloudvirt1036.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:02 wm-bot2: Testing wm-bot relay to #wikimedia-cloud-feed
* 17:55 wm-bot: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:54 wm-bot: Draining 'cloudvirt1036.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:04 wm-bot: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:03 wm-bot: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:03 wm-bot: Drained 'cloudvirt1034.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:51 wm-bot: Drained 'cloudvirt1033.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:37 wm-bot: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:37 wm-bot: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Draining 'cloudvirt1034.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 15:01 wm-bot: Drained 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 15:00 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:57 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:44 wm-bot: Drained 'cloudvirt1031.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:35 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:34 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:32 wm-bot: Drained 'cloudvirt1030.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:20 wm-bot: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:19 wm-bot: Draining 'cloudvirt1031.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:18 wm-bot: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:17 wm-bot: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:54 taavi: restart nova-fullstack on cloudcontrol1003 to pick up bastion ip change
* 13:43 wm-bot: Drained 'cloudvirt1029.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:23 wm-bot: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:22 wm-bot: Draining 'cloudvirt1029.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-22 ===
* 22:59 wm-bot: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:58 wm-bot: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-17 ===
* 01:09 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:53 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:52 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:52 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster

=== 2022-03-15 ===
* 20:58 wm-bot: Drained 'cloudvirt1026.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:14 wm-bot: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 13:14 wm-bot: Unsetting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 10:32 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 10:30 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo

=== 2022-03-14 ===
* 21:24 wm-bot: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:59 wm-bot: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:58 wm-bot: Setting cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:58 wm-bot: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:02 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:59 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:59 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:16 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:13 wm-bot: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:56 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:55 wm-bot: Setting cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:55 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:53 wm-bot: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:52 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:51 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:51 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:50 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 11:48 dcaro: rebased cookbooks on latest master, make sure you pull before sending new patches

=== 2022-03-08 ===
* 18:29 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:29 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:29 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:23 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:21 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:21 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:18 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:17 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:17 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:27 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:27 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:18 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:15 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:15 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:47 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:47 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:08 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:07 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:07 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:11 arturo: [codfw1dev] rebooting cloudservices servers for [[phab:T303179|T303179]]
* 13:07 arturo: [codfw1dev] rebooting cloudvirt servers for [[phab:T303179|T303179]]
* 13:06 arturo: [codfw1dev] rebooting cloudnet servers for [[phab:T303179|T303179]]
* 12:55 arturo: [codfw1dev] rebooting cloudcontrol servers for [[phab:T303179|T303179]]

=== 2022-03-03 ===
* 08:49 taavi: deploying cloudmetrics grafana to grafana 8, [[phab:T282863|T282863]]

=== 2022-03-02 ===
* 09:06 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/701347

=== 2022-02-28 ===
* 15:30 dcaro: cleaning up leftover snapshots from failed backups of the maps volume ([[phab:T302720|T302720]])

=== 2022-02-24 ===
* 17:04 andrewbogott: upgrading eqiad1 and codfw1dev to mariadb 10.5.15+maria~bullseye via 'apt-get install libmariadb3:amd64 galera-4 mariadb-server'
* 15:42 dcaro: stopping and starting mariadb on cloudcontrol1003 ([[phab:T302146|T302146]])
* 10:37 arturo: [codfw1dev] briefly installed galera-4 (26.4.11+1bullseye) over (26.4.9-0+deb11u1) on cloudcontrol2001-dev and then downgrade again to verify package install ([[phab:T302482|T302482]])

=== 2022-02-23 ===
* 20:39 taavi: added domain-wide 'designateadmin' and 'observer' roles to project-proxy-dns-manager service account [[phab:T295246|T295246]]
* 17:40 andrewbogott: restarting lots of openstack services to try to clear up the mess that is [[phab:T236101|T236101]]
* 12:13 arturo: cleaning up cinder volume snapshots, aborrero@cloudcontrol1005:~$ for i in $(sudo wmcs-openstack volume snapshot list -f value -c ID) ; do sudo wmcs-openstack volume snapshot delete $i ; done ([[phab:T302382|T302382]])
* 10:14 arturo: cleaning up neutron agents for non-existent servers cloudvirt100[1-9].eqiad.wmnet,cloudvirt10[12-15].eqiad.wmnet
* 10:05 dcaro: Deleting stuck novafullstack servers, to let the service create new ones ([[phab:T302369|T302369]])
* 09:56 arturo: neutron agent-delete bad663b3-fd25-4393-a546-{{Gerrit|4b1b4bdec4db}} (Linux bridge agent {{!}} cloudvirtan1001)
* 09:56 arturo: neutron agent-delete 1071c198-ed57-4b5a-9439-{{Gerrit|30e66a31aa69}} (Linux bridge agent {{!}} cloudvirtan1005)
* 09:55 arturo: neutron agent-delete 2eeef198-8af7-4e5d-bd73-{{Gerrit|e14a2a8d2404}} (Linux bridge agent {{!}} cloudvirtan1004)
* 09:55 arturo: neutron agent-delete afe173eb-35ba-444a-9960-{{Gerrit|899629786d2f}} (Linux bridge agent {{!}} cloudvirtan1003)
* 09:54 arturo: neutron agent-delete afcb9b7f-c1a6-4ff4-9b10-{{Gerrit|92bfbe8d1a56}} (Linux bridge agent {{!}} cloudvirtan1002)
* 09:39 dcaro: restarting neutron-api cloudcontrol1003 to see if the agent status update starts working ([[phab:T302369|T302369]])
* 09:38 dcaro: restarting neutron-dhcp-agent on cloudnet1003 ([[phab:T302369|T302369]])

=== 2022-02-22 ===
* 22:10 andrewbogott: raising project 'maps' quota by two tb -- [[phab:T300160|T300160]]
* 09:24 arturo: restarting mariadb @ cloudcontrol1003 ([[phab:T302146|T302146]])
* 09:13 arturo: restarting mariadb @ cloudcontrol1004 ([[phab:T302146|T302146]])

=== 2022-02-18 ===
* 21:57 andrewbogott: leaving cloudcontrol1003 downtimed with disabled puppet for the weekend.  Everything there should be stable and fine save rabbit which needs an upgrade.
* 21:30 andrewbogott: rebooting cloudcontrol1003 because rabbit is freaking out
* 17:25 andrewbogott: in-place upgrade of cloudcontrol1004 to bullseye -- [[phab:T281276|T281276]]
* 12:34 arturo: manually install prometheus-openstack-exporter on cloudcontrol1005 ([[phab:T302050|T302050]])

=== 2022-02-17 ===
* 23:02 andrewbogott: in-place upgrade to Bullseye on cloudcontrol1005 [[phab:T281276|T281276]]

=== 2022-02-15 ===
* 14:15 taavi: [codfw1dev] added domain-wide 'designateadmin' and 'observer' roles to codfw1dev-proxy-dns-manager service account [[phab:T295246|T295246]]

=== 2022-02-04 ===
* 10:12 arturo: restart backup_vms service in cloudvirt1024 ([[phab:T300956|T300956]])

=== 2022-02-03 ===
* 08:21 taavi: cloudmetrics1004: manually added an empty line to /etc/prometheus/blackbox.yml to make /usr/local/bin/blackbox-exporter-assemble happy (clearing &quot;performing a change every puppet run&quot; alert)

=== 2022-02-02 ===
* 02:36 andrewbogott: restarting mariadb on cloudcontrol1004

=== 2022-01-31 ===
* 10:15 arturo: cloudcontrol1005:~$ sudo systemctl restart backup_glance_images.service (failed state, no logs, icinga alert)

=== 2022-01-29 ===
* 18:24 taavi: delete 2 puppet prefixes in a weird state [[phab:T299750|T299750]]

=== 2022-01-27 ===
* 13:24 arturo: cloudmetrics1004:~ $ sudo systemctl restart wmcs_monitoring_graphite_rsync.service ([[phab:T300138|T300138]])

=== 2022-01-26 ===
* 19:09 andrewbogott: bootstrapping a fresh galera node on cloudcontrol1004
* 18:57 andrewbogott: restarting mariadb on cloudcontrol1004

=== 2022-01-25 ===
* 10:49 arturo: made cloudmetrics1001/1002 primary/backup respectively ([[phab:T299744|T299744]], [[phab:T297814|T297814]], [[phab:T300011|T300011]])

=== 2022-01-19 ===
* 16:38 andrewbogott: moving all scratch mounts to scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud

=== 2022-01-05 ===
* 03:11 andrewbogott: 'cp /etc/apt/sources.list /etc/apt/sources.list.prepuppet' on all VMs. Backing up state before puppetizing sources.list with https://gerrit.wikimedia.org/r/c/operations/puppet/+/751498

=== 2022-01-04 ===
* 12:44 dcaro: increasing the size_limit for labs ldap servers

=== 2021-12-26 ===
* 16:55 majavah: run attachLdapUser.php on wikitech for developer account &quot;Karthiksripal&quot;

=== 2021-12-24 ===
* 22:51 majavah: ran the wikireplica dns script on s5 [[phab:T298303|T298303]]

=== 2021-12-23 ===
* 21:42 majavah: deployed horizon wmf-proxy-dashboard update to fix editing of existing proxies

=== 2021-12-21 ===
* 10:39 arturo: dropped egress NAT exceptions for WMF apt repos, [[phab:T298042|T298042]]

=== 2021-12-15 ===
* 12:44 dcaro: Downtiming cloudvirt-wdqs1001 as it has no VMs running until disk space is fixed ([[phab:T297454|T297454]])

=== 2021-12-14 ===
* 10:26 dcaro: Moved the nova cache (/var/lib/nova/instances/_base) and the canary image local data (/var/lib/nova/instance/&lt;canary_image_id&gt;) to the root disk on cloudvirt-wdqs1001 to temporary free some space ([[phab:T297454|T297454]])

=== 2021-12-13 ===
* 18:08 wm-bot: Drained 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:50 wm-bot: Set cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:49 wm-bot: Setting cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:49 wm-bot: Draining 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:44 wm-bot: Drained 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:30 wm-bot: Set cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:30 wm-bot: Setting cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:30 wm-bot: Draining 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:13 wm-bot: Drained 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
* 16:50 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:47 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:47 wm-bot: Draining 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
* 16:44 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:43 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse

=== 2021-12-03 ===
* 18:56 andrewbogott: maintain-views and maintain-meta-p on clouddb1013-1020
* 10:49 majavah: deleting dbbackups-dashboard project [[phab:T296992|T296992]]

=== 2021-12-02 ===
* 01:17 wm-bot: Drained 'cloudvirt1028.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:56 wm-bot: Set cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:56 wm-bot: Setting cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:56 wm-bot: Draining 'cloudvirt1028.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:50 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:50 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:50 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:28 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:03 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:02 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:02 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster

=== 2021-12-01 ===
* 23:59 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:59 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 23:54 andrewbogott: *correction* adding spare cloudvirts 1044 and 1045 to the 'ceph' pool in order to make space for future juggling around [[phab:T296790|T296790]] and [[phab:T296792|T296792]]
* 23:53 andrewbogott: adding spare cloudvirts 1044 and 1055 to the 'ceph' pool in order to make space for future juggling around [[phab:T296790|T296790]] and [[phab:T296792|T296792]]

=== 2021-11-28 ===
* 17:48 andrewbogott: moved cloudvirt1018 out of the 'localstorage' aggregate and into 'maintenance' for [[phab:T296592|T296592]].  It will need to be moved back after the raid is rebuilt.

=== 2021-11-21 ===
* 07:19 dcaro_away: restarting designate-sink with some extra logs in it ([[phab:T296144|T296144]])

=== 2021-11-17 ===
* 15:48 andrewbogott: upgrading mariadb packages on eqiad1 cloudcontrols
* 15:39 andrewbogott: sudo cumin  &quot;cloud*&quot; 'apt-get update -y --allow-releaseinfo-change'
* 15:26 andrewbogott: updated mariadb packages on codfw1dev cloudcontrols to 1:10.3.31-0+deb10u1

=== 2021-11-12 ===
* 13:31 arturo: restarting glance-api services to make sure they work with new ceph auth creds ([[phab:T293752|T293752]])

=== 2021-11-08 ===
* 21:50 andrewbogott: returned clouddb pools back to normal after maintain_views run: https://gerrit.wikimedia.org/r/c/operations/puppet/+/737505 [[phab:T216481|T216481]]
* 20:07 andrewbogott: depooling clouddb1013 for maintain_views attempt
* 10:54 arturo: [codfw1dev] create service account `srv-networktests` following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for [[phab:T294955|T294955]]
* 10:34 arturo: create service account `srv-networktests` following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for [[phab:T294955|T294955]]

=== 2021-11-05 ===
* 11:18 wm-bot: Added 1 new OSDs ['cloudcephosd1024.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:17 wm-bot: Added OSD cloudcephosd1024.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:15 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:12 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:12 wm-bot: Adding OSD cloudcephosd1024.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:12 wm-bot: Adding new OSDs ['cloudcephosd1024.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance

=== 2021-11-04 ===
* 16:39 wm-bot: Added 1 new OSDs ['cloudcephosd1023.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:39 wm-bot: Added OSD cloudcephosd1023.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:37 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:34 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:33 wm-bot: Adding OSD cloudcephosd1023.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:33 wm-bot: Adding new OSDs ['cloudcephosd1023.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:17 wm-bot: Added 1 new OSDs ['cloudcephosd1022.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:17 wm-bot: Added OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:16 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:13 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:12 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:12 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:00 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:00 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:26 wm-bot: Added 1 new OSDs ['cloudcephosd1021.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:26 wm-bot: Added OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:23 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:20 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:19 wm-bot: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:19 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:16 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance

=== 2021-11-03 ===
* 17:22 arturo: [codfw1dev] installing keepalived 2.1.5 from buster-backports on cloudgw2001-dev/2002-dev ([[phab:T294956|T294956]])
* 11:45 arturo: [codfw1dev] downgrade kernel on cloudgw2001-dev/2002-dev ([[phab:T294853|T294853]], [[phab:T291813|T291813]])

=== 2021-11-02 ===
* 10:54 arturo: rebooting cloudnet1004/1003 for [[phab:T291813|T291813]]
* 10:43 arturo: [codfw1dev] rebooting cloudgw200[12]-dev for [[phab:T291813|T291813]]

=== 2021-10-24 ===
* 00:47 andrewbogott: deploying a change so that openstack clients use tls endpoints: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732738

=== 2021-10-21 ===
* 10:19 arturo: drop firewall exception on core routers for wiki replicas legacy setup ([[phab:T293897|T293897]])
* 10:12 arturo: drop NAT exception for wiki replicas legacy setup ([[phab:T293897|T293897]])

=== 2021-10-20 ===
* 21:06 andrewbogott: creating cloudinfra-nfs project [[phab:T293936|T293936]]

=== 2021-10-18 ===
* 19:21 andrewbogott: also ticked the 'admin' box on wikitech for majavah [[phab:T292827|T292827]]
* 18:58 andrewbogott: granting majavah 'admin' role in the 'admin' project and also in the default domain.  [[phab:T292827|T292827]]

=== 2021-10-14 ===
* 12:28 arturo: [codfw1dev] add DB grants for cloudbackup2002.codfw.wmnet IP address to the cinder DB ([[phab:T292546|T292546]])

=== 2021-10-13 ===
* 10:46 arturo: updating python3-neutron across the fleet ([[phab:T292936|T292936]])

=== 2021-10-12 ===
* 09:06 dcaro: upgrading eqiad cloudnet hosts neutron packages ([[phab:T292936|T292936]])
* 08:57 dcaro: upgrading codfw cloudnet hosts neutron packages ([[phab:T292936|T292936]])

=== 2021-10-05 ===
* 09:39 arturo: [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) [[phab:T291257|T291257]]

=== 2021-09-30 ===
* 14:50 andrewbogott: sudo cumin  &quot;cloud*&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot; and sudo cumin  &quot;lab*&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot;  [[phab:T292202|T292202]]
* 14:43 andrewbogott: ran sudo cumin --force --timeout 500 -o json  &quot;A:all&quot; &quot;ps -ef {{!}} grep nslcd &amp;&amp; service nslcd restart&quot; to get nslcd happy again [[phab:T292202|T292202]]

=== 2021-09-29 ===
* 09:41 arturo: [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured ([[phab:T291257|T291257]])

=== 2021-09-28 ===
* 16:23 bstorm: downtime for clouddb1020 to reduce re-pages in case this goes badly [[phab:T291963|T291963]]
* 16:21 bstorm: powering on clouddb1020 via remote console [[phab:T291963|T291963]]
* 15:58 bstorm: depooled clouddb1020 for repair [[phab:T291961|T291961]]
* 12:40 dcaro: Merged change on sssd for bullseye cloud hosts ([[phab:T291585|T291585]])
* 11:30 arturo: [codfw1dev] create floating IP 185.15.57.5 for manila-sharecontroller.cloudinfra-codfw1dev.codfw1dev.wmcloud.org ([[phab:T291257|T291257]])

=== 2021-09-27 ===
* 10:07 arturo: cloudcontrol1004 apparently healthy [[phab:T291446|T291446]]
* 09:25 arturo: rebooting cloudcontrol1004 for [[phab:T291446|T291446]]

=== 2021-09-24 ===
* 13:02 arturo: [codfw1dev] create VM manila-share-controller-01 on cloudinfra-codfw1dev
* 13:00 arturo: [codfw1dev] rebase labs/private.git on cloudinfra-puppetmaster-01, had merge conflict

=== 2021-09-21 ===
* 12:13 arturo: [codfw1dev] trying to create a manila service image ([[phab:T291257|T291257]])
* 11:45 arturo: [codfw1dev] created rabbitmq user ([[phab:T291257|T291257]])
* 11:32 arturo: [codfw1dev] populated manila DB &amp; created service endpoints ([[phab:T291257|T291257]])
* 11:06 arturo: [codfw1dev] give manila user admin role @ manila project ([[phab:T291257|T291257]])
* 11:06 arturo: [codfw1dev] created manila project ([[phab:T291257|T291257]])
* 10:57 arturo: [codfw1dev] created manila user @ labtestwikitech ([[phab:T291257|T291257]])
* 10:49 arturo: [codfw1dev] create manila database on cloudcontrol-dev nodes (galera) [[phab:T291257|T291257]]

=== 2021-09-20 ===
* 23:08 bstorm: ran `echo check &gt; /sys/block/md0/md/sync_action` on cloudcontrol1004 to check raid
* 22:48 andrewbogott: stopped puppet &amp; mariadb on cloudcontrol1004; it was flapping
* 22:44 andrewbogott: sudo touch /tmp/galera.disabled on cloudcontrol1004, the service seems troubled there
* 21:57 andrewbogott: moving cloudvirt1043 into the 'nfs' aggregate for [[phab:T291405|T291405]]

=== 2021-09-17 ===
* 11:35 arturo: [codfw1dev] install manila on cloudcontrol2001-dev ([[phab:T291257|T291257]])

=== 2021-09-16 ===
* 15:56 bstorm: removing downtime for labstore1005 so we'll know if it has another issue [[phab:T290318|T290318]]

=== 2021-09-09 ===
* 22:03 bstorm: restarted the prometheus-mysqld-exporter@s1 service as it was not working [[phab:T290630|T290630]]
* 03:15 bstorm: resetting swap on clouddb1017 [[phab:T290630|T290630]]
* 03:08 andrewbogott: stopping maintain-dbusers on labstore1004 for help diagnosing [[phab:T290630|T290630]]

=== 2021-09-03 ===
* 15:34 bstorm: rebooting labstore1005 to disconnect the drives from labstore1004 [[phab:T290318|T290318]]
* 15:24 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2002 [[phab:T290318|T290318]]
* 15:20 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2001 [[phab:T290318|T290318]]

=== 2021-08-30 ===
* 16:16 wm-bot: Added 1 new OSDs ['cloudcephosd1018.eqiad.wmnet'] - cookbook ran by andrew@buster
* 16:16 wm-bot: Added OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:10 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:07 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:07 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:07 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster

=== 2021-08-27 ===
* 18:57 andrewbogott: raising toolsbeta ram/core/instances quotas so majavah can experiment with bullseye

=== 2021-08-25 ===
* 14:45 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 14:42 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 14:42 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 14:42 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 14:41 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster

=== 2021-08-19 ===
* 17:39 bstorm: restarting glance image backup to try and clear the page

=== 2021-08-18 ===
* 16:21 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:21 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:21 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:17 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:16 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:15 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 14:47 andrewbogott: adding clouvirt1038 to the ceph aggregate, removing from the maintenance aggregate [[phab:T276922|T276922]]

=== 2021-08-17 ===
* 15:11 andrewbogott: rebooting cloudcephosd1008 to force raid rebuild -- [[phab:T287838|T287838]]

=== 2021-08-11 ===
* 13:51 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:48 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:47 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 13:47 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-08-10 ===
* 15:15 andrewbogott: restarting all designate services in eqiad1
* 15:04 andrewbogott: restarting designate-sink in eqiad1; it's complaining about rabbit but I don't want to restart rabbit yet

=== 2021-08-05 ===
* 09:37 dcaro: Taking one osd daemon down ot codfw cluster ([[phab:T288203|T288203]])

=== 2021-08-04 ===
* 19:20 bd808: Running deleteBatch.php on cloudweb2001-dev to remove legacy Heira: pages from labtestwiki

=== 2021-08-03 ===
* 17:40 bstorm: rerunning the glance backup script after failure

=== 2021-07-31 ===
* 00:10 andrewbogott: &quot;systemctl reset-failed cloud-init.service&quot; on all VMs for [[phab:T287309|T287309]]
* 00:08 andrewbogott: &quot;systemctl reset-failed cloud-final.service&quot; on all VMs for [[phab:T287309|T287309]]

=== 2021-07-27 ===
* 21:32 andrewbogott: putting cloudvirt1012 back into service [[phab:T286748|T286748]]
* 20:52 andrewbogott: draining VMs off of cloudvirt1012 so we can replace the battery for [[phab:T286748|T286748]]
* 15:15 andrewbogott: &quot;rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list&quot; cloud-wide

=== 2021-07-23 ===
* 15:22 bstorm: update wikireplicas-dns for s7 fix for web replicas

=== 2021-07-20 ===
* 17:07 andrewbogott: reloading haproxy on dbproxy1018 for [[phab:T286598|T286598]]
* 15:45 arturo: failback from labstore1006 to labstore1007 (dumps NFS) https://gerrit.wikimedia.org/r/c/operations/puppet/+/705417
* 00:10 bstorm: restarting nova-api on cloudcontrol1003 to try and recover whatever it's doing with designate_floating_ip_ptr_records_updater

=== 2021-07-19 ===
* 22:05 bstorm: set downtime scheduled for tomorrow from 1300 to 1600 UTC for cloudstore1008 and 1009 [[phab:T286599|T286599]]
* 20:40 andrewbogott: reloading haproxy on dbproxy1018 for [[phab:T286598|T286598]]
* 13:50 andrewbogott: upgrading mariadb to 10.3.29 on all cloudcontrols

=== 2021-07-16 ===
* 09:55 dcaro: checking HP raid issues on coludvirt1012 ([[phab:T286766|T286766]])

=== 2021-07-14 ===
* 21:08 andrewbogott: restarting lots of openstack services while trying to resolve [[phab:T286675|T286675]]
* 12:17 dcaro: doing ceph outage tests on codfw1 (fyi)

=== 2021-07-13 ===
* 10:57 dcaro: enabled autoscaling on codfw1 ceph cluster, setting a minimum of pgs on codfw1dev-compute to 128

=== 2021-07-02 ===
* 10:12 wm-bot: The cluster is not rebalance after adding the new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:12 wm-bot: Added 2 new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:12 wm-bot: Added OSD cloudcephosd1020.eqiad.wmnet... (2/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:10 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding OSD cloudcephosd1020.eqiad.wmnet... (2/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Added OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:02 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:02 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:01 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-07-01 ===
* 16:27 bstorm: failed over cloudstore1009 to cloudstore1008 [[phab:T224747|T224747]]
* 16:18 bstorm: downtimed cloudstore1008 and cloudstore1009 to fail over [[phab:T224747|T224747]]
* 14:25 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (2/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot: Added OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:20 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:20 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Added new OSD node cloudcephosd1016.eqiad.wmnet ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 11:13 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:58 dcaro: rebooting cloudcephosd1016 ([[phab:T285858|T285858]])
* 10:47 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:41 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-30 ===
* 21:48 bstorm: downtimed space alerts for scratch on cloudstore1008 until after the migration

=== 2021-06-25 ===
* 15:28 andrewbogott: restarting openstack services on cloudcontrol1005
* 09:16 arturo: icinga downtime cloudcontrols for 2h
* 08:20 dcaro: restarting rabbitmq on cloudcontrol100&lt;nowiki&gt;{&lt;/nowiki&gt;3,4&lt;nowiki&gt;}&lt;/nowiki&gt;

=== 2021-06-21 ===
* 13:54 dcaro: puppet fix merged and deployed, servers are back to normal
* 13:20 dcaro: merged broken puppet patch, downtimed all cloudvirts for 2h while fixing (nothing big, just added a bad systemd timer)

=== 2021-06-20 ===
* 22:21 andrewbogott: clearing admin-monitoring VMs; puppet has been failing lately due to a full drive on the puppetmaster

=== 2021-06-15 ===
* 01:18 bstorm: running a modified version of the prometheus dir size cron in screen [[phab:T284964|T284964]]

=== 2021-06-14 ===
* 10:13 dcaro: setting ssd to debug mode on tools-sgeexec-0917 ([[phab:T284130|T284130]])

=== 2021-06-10 ===
* 10:58 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:58 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:55 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:55 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:52 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:52 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:49 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:49 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:48 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:48 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:45 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:45 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:39 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:38 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:35 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:35 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:32 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:32 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:29 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:26 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:26 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:24 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:24 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-09 ===
* 17:33 arturo: removed icinga downtime for cloudmetrics1002 -- to see if hardware is healthy ([[phab:T281881|T281881]])
* 13:30 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:30 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus

=== 2021-06-08 ===
* 23:19 bd808: Downtimed cloudmetrics1002 in icinga until 2021-06-30 23:59:01 ([[phab:T281881|T281881]])
* 21:08 bstorm: downtiming grafana-labs for maintenance
* 16:28 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:27 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:24 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:24 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:22 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:21 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:18 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:17 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

=== 2021-06-07 ===
* 14:27 andrewbogott: moving cloudvirt1040 from 'maintenance' aggregate to 'ceph' aggregate  [[phab:T281399|T281399]]

=== 2021-06-01 ===
* 13:12 dcaro: Changed the ceph osd_memory_target on eqiad pool to 6Gi (we were reaching the limit, swapping at some points)
* 09:57 arturo: fix PTR record for 185.15.56.1 ([[phab:T284025|T284025]])
* 09:56 arturo: fix PTR record for 185.15.56.1 ([[phab:T248025|T248025]])

=== 2021-05-27 ===
* 14:58 wm-bot: Testing - cookbook ran by dcaro@vulcanus

=== 2021-05-26 ===
* 19:10 andrewbogott: reimaging cloudvirt1018 to support local VM storage
* 18:07 andrewbogott: draining cloudvirt1018, converting it to a local-storage  host like cloudvirt1019 and 1020 -- [[phab:T283296|T283296]]
* 14:36 dcaro: Enabled syslog logging for osd.55 on eqiad ceph cluster for testing ([[phab:T281247|T281247]])
* 14:36 dcaro: Enabled syslog logging on codfw ceph cluster (mon/osd/mgr) ([[phab:T281247|T281247]])
* 11:26 arturo: [codfw1dev] purge old kernel packages in cloudvirt200[12]-dev
* 11:03 arturo: created public flavor `g3.cores16.ram36.disk20` (even though it was requested as private in [[phab:T283293|T283293]], but may be useful for others)

=== 2021-05-25 ===
* 16:14 bd808: Closed #wikimedia-cloud-admin on f***node
* 16:11 bd808: Closed #wikimedia-cloud-feed on f***node
* 15:19 dcaro: rebooted cloudvirt1020, starting VMs ([[phab:T275893|T275893]])
* 15:13 dcaro: rebooting cloudvirt1020 ([[phab:T275893|T275893]])
* 14:42 dcaro: taking cloudvirt1020 out for maintenance (openstack wise) so no new VMs are scheduled on it ([[phab:T275893|T275893]])

=== 2021-05-24 ===
* 22:32 andrewbogott: changing the default ttl for eqiad1.wikimedia.cloud. from 3600 to 60; this should help us avoid madness when re-using hostnames.
* 11:20 arturo: created `g3.cores2.ram80.disk40.private` for the wmf-research-tools project, to allow resizing a 40G disk instance

=== 2021-05-22 ===
* 02:14 bstorm: downtiming SMART alerts on dumps server labstore1007 for the weekend because it has been flapping [[phab:T281045|T281045]]

=== 2021-05-13 ===
* 21:25 bstorm: converted the maps and scratch volumes on cloudstore1008 (standby) to drbd [[phab:T224747|T224747]]
* 15:45 bstorm: re-running wikireplicas-dns after refactor of config to make sure it doesn't change anything

=== 2021-05-12 ===
* 14:23 arturo: [codfw1dev] cleanup old unused agents (bgp, ovs)
* 11:37 arturo: [codfw1dev] replacing cloudnet2003-dev with cloudnet2004-dev ([[phab:T281381|T281381]])

=== 2021-05-11 ===
* 18:00 andrewbogott: adding 'trove' service project in advance of deploying trove in eqiad1
* 10:22 arturo: rebooted cloudgw1002 (active) thus causing a failover to cloudgw1001

=== 2021-05-09 ===
* 10:53 arturo: icinga-downtime cloudmetrics1002 for 3 months ([[phab:T275605|T275605]])

=== 2021-05-07 ===
* 13:51 andrewbogott: add inherited 'admin' right to novaadmin user throughout eqiad1.  I was trying to narrow down the rights here but lack of admin breaks some workflows, e.g. [[phab:T281894|T281894]] and [[phab:T282235|T282235]]

=== 2021-05-06 ===
* 15:31 arturo: about to migrating CloudVPS network to the cloudgw architecture [[phab:T270704|T270704]]
* 11:14 dcaro: restarting cinder-volume on the eqiad control nodes to refresh the ceph libraries ([[phab:T282109|T282109]])

=== 2021-05-05 ===
* 16:07 dcaro: disallowing insecure global ids on the eqiad ceph cluster ([[phab:T280641|T280641]])
* 15:15 wm-bot: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:11 wm-bot: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:11 wm-bot: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:40 wm-bot: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:39 wm-bot: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:14 wm-bot: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 14:10 wm-bot: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 12:35 wm-bot: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:31 wm-bot: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:13 wm-bot: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:47 wm-bot: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:47 wm-bot: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:21 wm-bot: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:21 wm-bot: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:45 wm-bot: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:45 wm-bot: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:02 wm-bot: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus

=== 2021-05-04 ===
* 16:05 wm-bot: Safe reboot of 'cloudvirt1028.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:45 wm-bot: Safe rebooting 'cloudvirt1028.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:44 wm-bot: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:22 wm-bot: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:19 wm-bot: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:15 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 13:19 dcaro: rebooting cloudmetrics1002, got stuck again ([[phab:T275605|T275605]])
* 10:04 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:10 wm-bot: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 09:10 wm-bot: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:34 wm-bot: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:20 wm-bot: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 08:03 wm-bot: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus

=== 2021-05-03 ===
* 23:53 bstorm: running `maintain-dbusers harvest-replicas` on labstore1004 [[phab:T281287|T281287]]
* 23:51 bstorm: running `maintain-dbusers harvest-replicas` on labstore1004
* 16:34 wm-bot: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 16:29 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:41 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:41 wm-bot: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T280641|T280641]]) - cookbook ran by dcaro@vulcanus
* 10:31 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T280641|T280641]] - cookbook ran by dcaro@vulcanus)
* 10:23 wm-bot: (from a cookbook)
* 09:12 dcaro: draining and rebooting coludvirt1021 ([[phab:T280641|T280641]])
* 08:26 dcaro: draining and rebooting coludvirt1018 ([[phab:T280641|T280641]])

=== 2021-04-30 ===
* 11:16 dcaro: draining and rebooting coludvirt1017, last one today ([[phab:T280641|T280641]])
* 10:37 dcaro: draining coludvirt1016 for reboot ([[phab:T280641|T280641]])
* 09:48 dcaro: draining coludvirt1013 for reboot ([[phab:T280641|T280641]])

=== 2021-04-29 ===
* 15:11 dcaro: hard rebooting cloudmetrics1002, got hung again ([[phab:T275605|T275605]])
* 07:53 dcaro: Upgrading ceph libraries on cloudcontrol1005 to octopus ([[phab:T274566|T274566]])
* 07:51 dcaro: Upgrading ceph libraries on cloudcontrol1003 to octopus ([[phab:T274566|T274566]])
* 07:50 dcaro: Upgrading ceph libraries on cloudcontrol1004 to octopus ([[phab:T274566|T274566]])

=== 2021-04-28 ===
* 21:11 andrewbogott: cleaning up more references to deleted hypervisors with delete from services where topic='compute' and version != 53;
* 20:48 andrewbogott: cleaning up references to deleted hypervisors with mysql:root@localhost [nova_eqiad1]&gt; delete from compute_nodes where hypervisor_version != '5002000';
* 19:40 andrewbogott: putting cloudvirt1040 into the maintenance aggregate pending more info about [[phab:T281399|T281399]]
* 18:11 andrewbogott: adding cloudvirt1040, 1041 and 1042 to the 'ceph' host aggregate -- [[phab:T275081|T275081]]
* 11:06 dcaro: All ceph server side upgraded to Octopus! \o/ ([[phab:T280641|T280641]])
* 10:57 dcaro: Got a PG getting stuck on 'remapping' after the OSD came up, had to unset the norebalance and then set it again to get it unstuck ([[phab:T280641|T280641]])
* 10:34 dcaro: Slow/blocked opns from cloudcephmon03, &quot;osd_failure(failed timeout osd.32...&quot; (cloudcephosd1005), unset the cluster noout/norebalance and went away in a few secs, setting it again and continuing... ([[phab:T280641|T280641]])
* 09:03 dcaro: Waiting for slow heartbeats from osd.58(cloudcephosd1002) to recover... ([[phab:T280641|T280641]])
* 08:59 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) all from osd.58, currently on cloudcephosd1002 ([[phab:T280641|T280641]])
* 08:58 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) all from osd.58 ([[phab:T280641|T280641]])
* 08:58 dcaro: During the upgrade, started getting warning 'slow osd heartbacks in the back', meaning that pings between osds are really slow (up to 190s) ([[phab:T280641|T280641]])
* 08:21 dcaro: Upgrading all the ceph osds on eqiad ([[phab:T280641|T280641]])
* 08:21 dcaro: The clock skew seems intermittent, there's another task to follw it [[phab:T275860|T275860]] ([[phab:T280641|T280641]])
* 08:18 dcaro: All equiad ceph mons and mgrs upgraded ([[phab:T280641|T280641]])
* 08:18 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, cloudcephmon1001, they are back ([[phab:T280641|T280641]])
* 08:15 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, it went away, I'm guessing systemd-timesyncd fixed it ([[phab:T280641|T280641]])
* 08:14 dcaro: During the upgrade, ceph detected a clock skew on cloudcephmon1002, looking ([[phab:T280641|T280641]])
* 07:58 dcaro: Upgrading ceph services on eqiad, starting with mons/managers ([[phab:T280641|T280641]])

=== 2021-04-27 ===
* 14:10 dcaro: codfw.openstack upgraded ceph libraries to 15.2.11 ([[phab:T280641|T280641]])
* 13:07 dcaro: codfw.openstack cloudvirt2002-dev done, taking cloudvirt2003-dev out to upgrade ceph libraries ([[phab:T280641|T280641]])
* 13:00 dcaro: codfw.openstack cloudvirt2001-dev back online, taking cloudvirt2002-dev out to upgrade ceph libraries ([[phab:T280641|T280641]])
* 10:51 dcaro: ceph.eqiad: cinder pool got it's pg_num increased to 1024, re-shuffle started ([[phab:T273783|T273783]])
* 10:48 dcaro: ceph.eqiad: Tweaked the target_size_ratio of all the pools, enabling autoscaler (it will increase cinder pool only) ([[phab:T273783|T273783]])
* 09:14 dcaro: manually force stopping the server puppetmaster-01 to unblock migration (in codfw1)
* 09:14 dcaro: manually force stopping the server puppetmaster-01 to unblock migration
* 08:59 dcaro: manually force stopping the server exploding-head on codfw, to try cold migration
* 08:47 dcaro: restarting nova-compute on cloudvirt2001-dev after upgrading ceph libraries to 15.2.11

=== 2021-04-26 ===
* 20:56 andrewbogott: deleting spurious 'codfw1dev' and 'codw1dev-4' regions in the dallas deployment; regions without endpoints break a bunch of things
* 09:45 dcaro: draining cloudvirt2001-dev with the new cookbooks ([[phab:T280641|T280641]])

=== 2021-04-23 ===
* 13:49 dcaro: testing the drain_cloudvirt cookbook on codfw1 openstack cluster, draining cloudvirt2001 ([[phab:T280641|T280641]])
* 11:12 dcaro: testing the drain_cloudvirt cookbook on codfw1 openstack cluster ([[phab:T280641|T280641]])
* 09:32 dcaro: finished upgrade of ceph cluster on codfw1 using exclusively cookbooks ([[phab:T280641|T280641]])
* 09:17 dcaro: testing the upgrade_osds cookbook on codfw1 ceph cluster ([[phab:T280641|T280641]])
* 08:17 dcaro: testing the upgrade_mons cookbook on codfw1 ceph cluster ([[phab:T280641|T280641]])

=== 2021-04-21 ===
* 17:59 dcaro: all monitors upgraded on codfw1 with one cookbook `cookbook --verbose -c ~/.config/spicerack/cookbook.yaml wmcs.ceph.upgrade_mons --monitor-node-fqdn cloudcephmon2002-dev.codfw.wmnet` ([[phab:T280641|T280641]])
* 17:47 dcaro: upgrading monitors and mrg nodes on codfw ceph cluster ([[phab:T280641|T280641]])
* 13:26 dcaro: testing ceph upgrade cookbook on cloudcephmon2002-dev ([[phab:T280641|T280641]])

=== 2021-04-20 ===
* 20:21 andrewbogott: reboot cloudservices1003
* 20:13 andrewbogott: reboot cloudservices1004

=== 2021-04-19 ===
* 08:40 dcaro: enabling puppet on labstore1004 after mysql restart ([[phab:T279657|T279657]])
* 08:09 dcaro: downtiming labstore1004 and stopping puppet for mysql restart ([[phab:T279657|T279657]])

=== 2021-04-14 ===
* 10:48 dcaro: Upgrade of codfw ceph to octopus 15.2.20 done, will run some performance tests now ([[phab:T274566|T274566]])
* 10:41 dcaro: Upgrade of codfw ceph to octopus 15.2.20, mgrs upgraded, osds next ([[phab:T274566|T274566]])
* 10:37 dcaro: Upgrade of codfw ceph to octopus 15.2.20, mons upgraded, mgrs next ([[phab:T274566|T274566]])
* 10:15 dcaro: starting the upgrade of codfw ceph to octopus 15.2.20 ([[phab:T274566|T274566]])
* 10:07 dcaro: Merged the ceph 15 (Octopus) repo deployment to codfw, only the repo, not the packages ([[phab:T274566|T274566]])

=== 2021-04-13 ===
* 16:42 dcaro: Ceph balancer got the cluster to eval 0.014916, that is 88-77% usage for compute pool, and 28-19% usage for the cinder one \o/ ([[phab:T274573|T274573]])
* 15:08 dcaro: Activating continuous upmap balancer, keeping a close eye ([[phab:T274573|T274573]])
* 15:03 dcaro: Executing a second pass, there's still movements to improve the eval of 0.030075 ([[phab:T274573|T274573]])
* 15:02 dcaro: First pass finished, improved eval to 0.030075 ([[phab:T274573|T274573]])
* 14:49 dcaro: Running the first_pass balancing plan on ceph eqiad, current eval 0.030622 ([[phab:T274573|T274573]])
* 14:43 dcaro: enabling ceph upmap pg balancer on equiad ([[phab:T274573|T274573]])
* 14:36 andrewbogott: upgrading codfw1dev to version Victoria, [[phab:T261137|T261137]]
* 13:11 andrewbogott: upgrading eqiad1 designate to version Victoria, [[phab:T261137|T261137]]
* 10:44 dcaro: enabled ceph upmap balancer on codfw ([[phab:T274573|T274573]],[[phab:T274573|T274573]])

=== 2021-04-07 ===
* 21:33 andrewbogott: upgrading codfw1dev designate to Victoria

=== 2021-04-04 ===
* 17:36 andrewbogott: upgrading eqiad1 designate to Ussuri

=== 2021-04-02 ===
* 14:12 andrewbogott: upgrading codfw1dev to OpenStack version Ussuri

=== 2021-04-01 ===
* 12:15 dcaro: Restoring the 4.9 kernel on cloudcephosd2003-dev and upgrading ([[phab:T274565|T274565]])
* 10:29 dcaro: Done restoring the 4.9 kernel on cloudcephosd2001-dev and upgrading, requires logging into console to boot from the older kernel before removing the newer one ([[phab:T274565|T274565]])
* 10:10 dcaro: Restoring the 4.9 kernel on cloudcephosd2001-dev and upgrading ([[phab:T274565|T274565]])

=== 2021-03-31 ===
* 08:47 dcaro: upgrading cinder on codfw cloudcontrol2* nodes ([[phab:T278845|T278845]])

=== 2021-03-30 ===
* 09:53 arturo: rebooting cloudnet1003 to cleanup conntrack table, it wouldn't cleanup by hand ...

=== 2021-03-28 ===
* 15:42 andrewbogott: updated debian-10.0-buster base image

=== 2021-03-27 ===
* 09:54 arturo: cleanup conntrack table in qrouter nents in cloudnet1003 (backup)

=== 2021-03-25 ===
* 19:03 andrewbogott: deleting all unused (per wmcs-imageusage) Jessie base images from Glance
* 17:15 andrewbogott: refreshing puppet compiler facts for tools project
* 10:31 dcaro: kernel upgrade on osds on codfw done, running performance tests ([[phab:T274565|T274565]])
* 10:24 dcaro: upgrading kernel on cloudcephosd2003-dev and reboot ([[phab:T274565|T274565]])
* 10:18 dcaro: upgrading kernel on cloudcephosd2002-dev and reboot ([[phab:T274565|T274565]])
* 10:08 dcaro: upgrading kernel on cloudcephmon2003-dev and reboot ([[phab:T274565|T274565]])

=== 2021-03-24 ===
* 09:19 dcaro: restarted wmcs-backup on cloudvirt1024 as it failed due to an image being removed while running ([[phab:T276892|T276892]])

=== 2021-03-23 ===
* 11:33 arturo: root@cloudcontrol1005:~# wmcs-novastats-dnsleaks --delete

=== 2021-03-22 ===
* 10:10 arturo: cleanup conntrack table in standby node: aborrero@cloudnet1003:~ $ sudo ip netns exec qrouter-d93771ba-2711-4f88-804a-{{Gerrit|8df6fd03978a}} conntrack -F

=== 2021-03-19 ===
* 17:18 bstorm: running `ALTER TABLE account MODIFY COLUMN type ENUM('user','tool','paws');` against the labsdbaccounts database on m5 [[phab:T276284|T276284]]
* 14:29 andrewbogott: switching admin-monitoring project to use an upstream debian image; I want to see how this affects performance
* 00:30 bstorm: downtimed labstore1004 to check some things in debug mode

=== 2021-03-17 ===
* 17:28 bstorm: restarted the backup-glance-images job to clear errors in systemd [[phab:T271782|T271782]]
* 17:16 andrewbogott: set default cinder quota for projects to 80Gb with &quot;update quota_classes set hard_limit=80 where resource='gigabytes';&quot; on database 'cinder'
* 16:58 andrewbogott: disabling all flavors with &gt;20Gb root storage with &quot;update flavors set disabled=1 where root_gb&gt;20;&quot; in nova_eqiad1_api

=== 2021-03-10 ===
* 16:51 arturo: rebooting cloudvirt1030 for [[phab:T275753|T275753]]
* 13:14 dcaro: starting manually the canary VM for cloudvirt1029 (nova start 349830f6-3b39-4a8c-ada4-{{Gerrit|a7439f65cffe}}) ([[phab:T275753|T275753]])
* 12:51 arturo: draining cloudvirt1030 for [[phab:T275753|T275753]]
* 12:47 arturo: rebooting cloudvirt1029 for [[phab:T275753|T275753]]
* 11:56 arturo: [codfw1dev] restart rabbitmq-server in all 3 cloudcontrol servers for [[phab:T276964|T276964]]
* 11:53 arturo: [codfw1dev] restart nova-conductor in all 3 cloudcontrol servers for [[phab:T276964|T276964]]
* 11:31 arturo: draining cloudvirt1029 for [[phab:T275753|T275753]]
* 11:29 arturo: rebooting cloudvirt1013 for [[phab:T275753|T275753]]
* 11:05 arturo: draining cloudvirt1013 for [[phab:T275753|T275753]]
* 11:00 arturo: rebooting cloudvirt1028 for [[phab:T275753|T275753]]
* 10:33 arturo: draining cloudvirt1028 for [[phab:T275753|T275753]]
* 10:29 arturo: rebooting cloudvirt1023 for [[phab:T275753|T275753]]
* 09:37 arturo: draining cloudvirt1023 for [[phab:T275753|T275753]]
* 09:07 arturo: [codfw1dev] reimaging cloudvirt2003-dev ([[phab:T276964|T276964]])

=== 2021-03-09 ===
* 16:27 arturo: rebooting cloudvirt1027 ([[phab:T275753|T275753]])
* 13:39 arturo: draining cloudvrit1027 for [[phab:T275753|T275753]]
* 13:35 arturo: icinga-downtime cloudvirt1038 for 30 days for [[phab:T276922|T276922]]
* 13:21 arturo: add cloudvirt1039 to the ceph host aggregate (no longer a spare, we have cloudvirt1038 with HW failures)
* 12:52 arturo: cloudvirt1038 hard powerdown / powerup for [[phab:T276922|T276922]]
* 12:33 arturo: rebooting cloudvirt1038 ([[phab:T275753|T275753]])
* 10:58 arturo: draining cloudvirt1038 ([[phab:T275753|T275753]])
* 10:54 arturo: rebooting cloudvirt1037 ([[phab:T275753|T275753]])
* 09:59 arturo: draining cloudvirt1037 ([[phab:T275753|T275753]])
* 09:12 dcaro: restarted the wmcs-backup service on cloudvirt1024 to retry the backups (failed because a VM was removed in-between, [[phab:T276892|T276892]])

=== 2021-03-05 ===
* 21:40 andrewbogott: replacing 'observer' role with 'reader' role in eqiad1 [[phab:T276018|T276018]]
* 21:21 andrewbogott: replacing 'observer' role with 'reader' role in eqiad1
* 16:23 arturo: rebooting cloudvirt1036 for [[phab:T275753|T275753]]
* 12:30 arturo: draining cloudvirt1036 for [[phab:T275753|T275753]]
* 12:25 arturo: rebooting cloudvirt1035 for [[phab:T275753|T275753]]
* 10:49 arturo: rebooting cloudvirt1035 for [[phab:T275753|T275753]]
* 10:47 arturo: rebooting cloudvirt1034 for [[phab:T275753|T275753]]
* 10:26 arturo: draining cloudvirt1034 for [[phab:T275753|T275753]]
* 10:25 arturo: rebooting cloudvirt1033 for [[phab:T275753|T275753]]
* 09:18 arturo: draining cloudvirt1033 for [[phab:T275753|T275753]]

=== 2021-03-04 ===
* 18:36 andrewbogott: rebooting cloudmetrics1002; the console is hanging
* 16:59 arturo: rebooting cloudvirt1032 for [[phab:T275753|T275753]]
* 16:34 arturo: draining cloudvirt1032 for [[phab:T275753|T275753]]
* 16:33 arturo: rebooting cloudvirt1031 for [[phab:T275753|T275753]]
* 16:11 arturo: draining cloudvirt1031 for [[phab:T275753|T275753]]
* 16:09 arturo: rebooting cloudvirt1026 for [[phab:T275753|T275753]]
* 15:57 arturo: draining cloudvirt1026 for [[phab:T275753|T275753]]
* 15:55 arturo: rebooting cloudvirt1025 for [[phab:T275753|T275753]]
* 15:41 arturo: draining cloudvirt1025 for [[phab:T275753|T275753]]
* 15:12 arturo: rebooting cloudvirt1024 for [[phab:T275753|T275753]]
* 11:29 arturo: draining cloudvirt1024 for [[phab:T275753|T275753]]
* 11:24 dcaro: rebooted cloudvirt1022, re-adding to ceph and removing from maintenance host aggregate for [[phab:T275753|T275753]]
* 11:01 dcaro: rebooting cloudvirt1022 for [[phab:T275753|T275753]]
* 09:12 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]

=== 2021-03-03 ===
* 17:16 andrewbogott: restarting rabbitmq-server on cloudcontrol1003,1004,1005; trying to explain amqp errors in scheduler logs
* 16:03 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]
* 16:03 dcaro: draining cloudvirt1022 for [[phab:T275753|T275753]]
* 16:00 arturo: move cloudvirt1013 into the 'toobusy' host aggregate, it has 221% cpu subscription and 82% MEM subscription
* 15:34 arturo: rebooting cloudvirt1021 for [[phab:T275753|T275753]]
* 14:31 arturo: draining cloudvirt1021 for [[phab:T275753|T275753]]
* 13:59 arturo: rebooting cloudvirt1018 for [[phab:T275753|T275753]]
* 13:28 arturo: draining cloudvirt1018 for [[phab:T275753|T275753]]
* 12:49 arturo: rebooting cloudvirt1017 for [[phab:T275753|T275753]]
* 12:22 arturo: draining cloudvirt1017 for [[phab:T275753|T275753]]
* 12:20 arturo: rebooting cloudvirt1016 for [[phab:T275753|T275753]]
* 12:01 arturo: draining cloudvirt1016 for [[phab:T275753|T275753]]
* 11:59 arturo: cloudvirt1014 now in the ceph host aggregate
* 11:58 arturo: rebooting cloudvirt1014 for [[phab:T275753|T275753]]
* 11:50 arturo: moved cloudvirt1023 away from the maintenance host aggregate, leave it in the ceph aggregate (was in the 2)
* 11:47 arturo: moved cloudvirt1014 to the 'maintenance' host aggregate, drain it for [[phab:T275753|T275753]]
* 10:01 arturo: icinga-downtime cloudnet1003 for 14 days bc potential alerting storm due to firmware issues ([[phab:T271058|T271058]])
* 10:01 arturo: rebooting again cloudnet1003 (no network failover) ([[phab:T271058|T271058]])
* 09:59 arturo: update firmware-bnx2x from 20190114-2 to 20200918-1~bpo10+1 on cloudnet1003 ([[phab:T271058|T271058]])
* 09:30 arturo: installing linux kernel 5.10.13-1~bpo10+1 in cloudnet1003 and rebooting it (network failover) ([[phab:T271058|T271058]])

=== 2021-03-02 ===
* 17:16 andrewbogott: rebooting cloudvirt1039 to see if I can trigger [[phab:T276208|T276208]]
* 16:10 arturo: [codfw1dev] restart nova-compute on cloudvirt2002-dev
* 11:59 arturo: moved cloudvirt1012 to 'maintenance' host aggregate. Drain it with `wmcs-drain-hypervisor` to reboot it for [[phab:T275753|T275753]]
* 11:59 arturo: cloudvirt1023 is affected by [[phab:T276208|T276208]] and cannot be rebooted. Put it back into the ceph hos aggregate
* 10:43 arturo: moved cloudvirt1013 cloudvirt1032 cloudvirt1037 back into the 'ceph' host aggregate
* 10:13 arturo: moved cloudvirt1023 to 'maintenance' host aggregate. Drain it with `wmcs-drain-hypervisor` to reboot it for [[phab:T275753|T275753]]

=== 2021-03-01 ===
* 20:12 andrewbogott: removing novaadmin from all projects save 'admin' for [[phab:T274385|T274385]]
* 19:51 andrewbogott: removing novaobserver from all projects save 'observer' for [[phab:T274385|T274385]]
* 19:50 andrewbogott: adding inherited domain-wide roles to novaadmin and novaobserver as per [[phab:T274385|T274385]]

=== 2021-02-28 ===
* 04:54 andrewbogott: restarted redis-server on tools-redis-1003 and tools-redis-1004 in an attempt to reduce replag, no real change detected

=== 2021-02-27 ===
* 00:33 andrewbogott: sudo cumin --timeout 500 &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i buster &amp;&amp; uname -r  {{!}} grep -v 4.19.0-14-amd64 &amp;&amp; reboot'
* 00:28 andrewbogott: sudo cumin --timeout 500 &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i buster &amp;&amp; uname -r  {{!}} grep -v 4.19.0-14-amd64 &amp;&amp; echo reboot'
* 00:09 andrewbogott: sudo cumin &quot;A:all and not O&lt;nowiki&gt;{&lt;/nowiki&gt;project:clouddb-services&lt;nowiki&gt;}&lt;/nowiki&gt;&quot; 'lsb_release -c {{!}} grep -i stretch &amp;&amp; uname -r  {{!}} grep -v 4.19.0-0.bpo.14-amd64 &amp;&amp; reboot'

=== 2021-02-26 ===
* 14:58 dcaro: [eqiad] rebooting cloudcephosd1015 (last osd \o/) for kernel upgrade ([[phab:T275753|T275753]])
* 14:51 dcaro: [eqiad] rebooting cloudcephosd1014 for kernel upgrade ([[phab:T275753|T275753]])
* 14:44 dcaro: [eqiad] rebooting cloudcephosd1013 for kernel upgrade ([[phab:T275753|T275753]])
* 14:38 dcaro: [eqiad] rebooting cloudcephosd1012 for kernel upgrade ([[phab:T275753|T275753]])
* 14:31 dcaro: [eqiad] rebooting cloudcephosd1011 for kernel upgrade ([[phab:T275753|T275753]])
* 14:25 dcaro: [eqiad] rebooting cloudcephosd1010 for kernel upgrade ([[phab:T275753|T275753]])
* 14:17 dcaro: [eqiad] rebooting cloudcephosd1009 for kernel upgrade ([[phab:T275753|T275753]])
* 13:54 dcaro: [eqiad] downtimed alert1001 Ceph OSDs down alert until 18:00 GMT+1 as that is not under the host being rebooted ([[phab:T275753|T275753]])
* 13:51 dcaro: [eqiad] rebooting cloudcephosd1008 for kernel upgrade ([[phab:T275753|T275753]])
* 13:45 dcaro: [eqiad] rebooting cloudcephosd1007 for kernel upgrade ([[phab:T275753|T275753]])
* 13:38 dcaro: [eqiad] rebooting cloudcephosd1006 for kernel upgrade ([[phab:T275753|T275753]])
* 12:07 dcaro: [eqiad] rebooting cloudcephosd1005 for kernel upgrade ([[phab:T275753|T275753]])
* 12:00 arturo: rebooting cloudcontrol1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:42 arturo: rebooting cloudcontrol1004 for kernel upgrade ([[phab:T275753|T275753]])
* 11:41 dcaro: [eqiad] rebooting cloudcephosd1004 for kernel upgrade ([[phab:T275753|T275753]])
* 11:32 dcaro: [eqiad] rebooting cloudcephosd1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:30 arturo: rebooting cloudcontrol1005 for kernel upgrade ([[phab:T2|T2]]
* 11:26 dcaro: [eqiad] rebooting cloudcephosd1002 for kernel upgrade ([[phab:T275753|T275753]])
* 11:16 dcaro: [eqiad] rebooting cloudcephosd1001 for kernel upgrade ([[phab:T275753|T275753]])
* 11:11 dcaro: [eqiad] rebooting cloudcephmon1003 for kernel upgrade ([[phab:T275753|T275753]])
* 11:05 dcaro: [eqiad] rebooting cloudcephmon1002 for kernel upgrade ([[phab:T275753|T275753]])
* 10:59 dcaro: [eqiad] rebooting cloudcephmon1001 for kernel upgrade ([[phab:T275753|T275753]])
* 10:45 arturo: rebooting cloudvirt1039 into a new kernel ([[phab:T275753|T275753]]) --- spare
* 10:43 dcaro: [codfw1dev] rebooting cloudcephmon2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:38 dcaro: [codfw1dev] rebooting cloudcephmon2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:29 dcaro: [codfw1dev] rebooting cloudcephmon2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:24 arturo: [codfw1dev] purge old kernel packages on cloudvirt2003-dev to force boot into a new kernel ([[phab:T275753|T275753]])
* 10:11 arturo: [codfw1dev] manually creating /boot/grub/ on cloudvirt2003-dev to allow update-grub2 to run (so it can reboot into a new kernel) ([[phab:T275753|T275753]])
* 10:11 dcaro: [codfw1dev] rebooting cloudcephosd2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:05 dcaro: [codfw1dev] rebooting cloudcephosd2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 10:01 arturo: [codfw1dev] rebooting cloudvirt200X-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:59 arturo: [codfw1dev] rebooting cloudweb2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:53 arturo: [codfw1dev] rebooting cloudservices2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:51 arturo: [codfw1dev] rebooting cloudservices2002-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:45 arturo: [codfw1dev] rebooting cloudcontrol2004-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:44 arturo: [codfw1dev] rebooting cloudbackup[2001-2002].codfw.wmnet for kernel upgrade ([[phab:T275753|T275753]])
* 09:43 dcaro: [codfw1dev] rebooting cloudcephosd2001-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:41 arturo: [codfw1dev] rebooting cloudcontrol2003-dev for kernel upgrade ([[phab:T275753|T275753]])
* 09:33 arturo: [codfw1dev] rebooting cloudcontrol2001-dev for kernel upgrade ([[phab:T275753|T275753]])

=== 2021-02-25 ===
* 14:56 arturo: deployed wmcs-netns-events daemon to all cloudnet servers ([[phab:T275483|T275483]])

=== 2021-02-24 ===
* 11:07 arturo: force-reboot cloudmetrics1002, add icinga downtime for 2 hours. Investigating some server issue
* 00:17 bstorm: set  --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main stretch image in glance on eqiad1 [[phab:T275430|T275430]]

=== 2021-02-23 ===
* 22:43 bstorm: set  --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main buster image in glance on eqiad1 [[phab:T275430|T275430]]
* 20:36 andrewbogott: adding r/o access to the eqiad1-glance-images ceph pool for the client.eqiad1-compute for [[phab:T275430|T275430]]
* 10:49 arturo: rebooting clounet1004 into new kernel from buster-bpo ([[phab:T271058|T271058]])
* 10:49 arturo: installing linux-image-amd64 from buster-bpo 5.10.13-1~bpo10+1 in cloudnet1004 ([[phab:T271058|T271058]])

=== 2021-02-22 ===
* 17:15 bstorm: restarting nova-compute on cloudvirt1016 and cloudvirt1036 in case it helps [[phab:T275411|T275411]]
* 15:02 dcaro: Re-uploaded the debian buster 10.0 image from rbd to glance, that worked, re-spawning all the broken instances ([[phab:T275378|T275378]])
* 11:12 dcaro: Refreshing all the canary instances ([[phab:T275354|T275354]])

=== 2021-02-18 ===
* 14:50 arturo: rebooting cloudnet1004 for [[phab:T271058|T271058]]
* 10:25 dcaro: Rebooting cloudmetrics1001 to apply new kernel ([[phab:T275116|T275116]])
* 10:16 dcaro: Rebooting cloudmetrics1002 to apply new kernel ([[phab:T275116|T275116]])
* 10:14 dcaro: Upgrading grafana on cloudmetrics1002 ([[phab:T275116|T275116]])
* 10:12 dcaro: Upgrading grafana on cloudmetrics1001 ([[phab:T275116|T275116]])

=== 2021-02-17 ===
* 15:58 arturo: deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/664845 to cloudnet servers ([[phab:T268335|T268335]])

=== 2021-02-15 ===
* 16:25 arturo: [codfw1dev] rebooting all cloudgw200x-dev / cloudnet200x-dev servers ([[phab:T272963|T272963]])
* 15:45 arturo: [codfw1dev] drop subnet definition for cloud-instances-transport1-b-codfw ([[phab:T272963|T272963]])
* 15:45 arturo: [codfw1dev] connect virtual router cloudinstances2b-gw to vlan cloud-gw-transport-codfw (185.15.57.10) ([[phab:T272963|T272963]])

=== 2021-02-11 ===
* 12:01 arturo: [codfw1dev] drop instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` (was buster, cannot use it yet)
* 11:59 arturo: [codfw1dev] create instance `tools-codfw1dev-bastion-2` (stretch) in `tools-codfw1dev` to test stuff related to [[phab:T272397|T272397]]
* 11:45 arturo: [codfw1dev] create instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` to test stuff related to [[phab:T272397|T272397]]
* 11:42 arturo: [codfw1dev] drop `tools` project, create `tools-codfw1dev`
* 11:38 arturo: [codfw1dev] drop `coudinfra` project (we are using `cloudinfra-codfw1dev` there)
* 05:37 bstorm: downtimed cloudnet1004 for another week [[phab:T271058|T271058]]

=== 2021-02-09 ===
* 15:23 arturo: icinga-downtime for 2h everything *labs *cloud for openstack upgrades
* 11:14 dcaro: Merged the osd scheduler change for all osds, applying on all cloudcephosd* ([[phab:T273791|T273791]])

=== 2021-02-08 ===
* 18:50 bstorm: enabled puppet on cloudvirt1023 for now [[phab:T274144|T274144]]
* 18:44 bstorm: restarted the backup_vms.service on cloudvirt1027 [[phab:T274144|T274144]]
* 17:51 bstorm: deleted project pki [[phab:T273175|T273175]]

=== 2021-02-05 ===
* 10:59 arturo: icinga-downtime labstore1004 tools share space check for 1 week ([[phab:T272247|T272247]])
* 10:21 dcaro: This was affecting maps and several others, maps and project-proxy have been fixed ([[phab:T273956|T273956]])
* 09:19 dcaro: Some certs around the infra are expired ([[phab:T273956|T273956]])

=== 2021-02-04 ===
* 10:12 dcaro: Increasing the memory limit of osds in eqiad from 8589934592(8G) to 12884901888(12G) ([[phab:T273851|T273851]])

=== 2021-02-03 ===
* 09:59 dcaro: Doing a full vm backup on cloudvirt1024 with the new script ([[phab:T260692|T260692]])
* 01:50 bstorm: icinga-downtime cloudnet1004 for a week [[phab:T271058|T271058]]

=== 2021-02-02 ===
* 17:14 dcaro: Changed osd memory limit from 4G to 8G ([[phab:T273649|T273649]])
* 11:00 arturo: icinga-downtime cloudvirt-wdqs1001 for 1 week ([[phab:T273579|T273579]])
* 03:12 andrewbogott: running /usr/local/sbin/wmcs-purge-backups and /usr/local/sbin/wmcs-backup-instances on cloudvirt1024 to see why the backup job paged

=== 2021-01-29 ===
* 15:36 andrewbogott: disabling puppet and some services on eqiad1 cloudcontrol nodes; replacing nova-placement-api with placement-api

=== 2021-01-28 ===
* 19:44 andrewbogott: shutting down cloudcontrol2001-dev because it's in a partially upgraded state; will revive when it's time for Train

=== 2021-01-27 ===
* 00:50 bstorm: icinga-downtime cloudnet1004 for a week [[phab:T271058|T271058]]

=== 2021-01-22 ===
* 16:44 andrewbogott: upgrading designate on cloudvirt1003/1004 to OpenStack 'train'
* 11:29 dcaro: Doing some tests removed cloudcontrol1003 puppet cert, regenerating...

=== 2021-01-21 ===
* 11:35 arturo: merging core router firewall changes https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657439 ([[phab:T209082|T209082]])
* 11:30 arturo: merging core router firewall changes https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657358 ([[phab:T272486|T272486]], [[phab:T209082|T209082]])

=== 2021-01-20 ===
* 10:49 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657302 ([[phab:T209082|T209082]])
* 10:05 dcaro: Everything looks ok, created a new vm with a volume in ceph without issues, and on warnings/errors on ceph status, closing ([[phab:T272303|T272303]])
* 09:55 dcaro: Eqiad ceph cluster uprgaded, doing sanity checks ([[phab:T272303|T272303]])
* 09:46 dcaro: 75% of the eqiad cluster upgraded... continuing ([[phab:T272303|T272303]])
* 09:37 dcaro: 25% of the eqiad cluster upgraded... continuing ([[phab:T272303|T272303]])
* 09:24 dcaro: Mgr daemons upgraded and running, upgrading osd daemons on servers cloudcephosd1*, this make take a bit longer ([[phab:T272303|T272303]])
* 09:22 dcaro: Mon daemons upgraded and running, upgrading mgr daemons on servers cloudcephmon1* ([[phab:T272303|T272303]])
* 09:16 dcaro: Starting eqiad ceph upgrade, upgrading the mon servers cloudcephmon1* ([[phab:T272303|T272303]])
* 09:01 dcaro: Will start the ceph upgrade in 15 min, no downtime nor performance impact is expected ([[phab:T272303|T272303]])

=== 2021-01-19 ===
* 10:17 arturo: icinga-downtime cloudnet1004 for 1 week ([[phab:T271058|T271058]])

=== 2021-01-18 ===
* 16:00 dcaro: Codfw1 ceph cluster uprgaded, will wait until tomorrow to see if there's any instability, but everything looks fine ([[phab:T272303|T272303]])
* 15:38 dcaro: Upgraded mgr sevices on codfw ceph cluster, starting with osd ones ([[phab:T272303|T272303]])
* 15:35 dcaro: Upgraded mon sevices on codfw ceph cluster, starting with mgr ones ([[phab:T272303|T272303]])
* 15:21 dcaro: Starting upgrade of ceph mon nodes on codfw ([[phab:T272303|T272303]])
* 15:06 dcaro: re-enabling puppet on cloudcephosd2* hosts
* 13:53 dcaro: disabling puppet on cloudcephosd2* to resume perf tests
* 10:50 dcaro: re-enabling puppet on cephcloudosd2* (codfw)
* 10:07 dcaro: disabling puppet on cephcloudosd2* (codfw) to do some performance tests
* 09:00 dcaro: Enabling custom application 'cinder' on pool codfw1dev-cinder to get rid of health warnings

=== 2021-01-17 ===
* 16:53 arturo: icinga downtime labstore1004 /srv/tools space check for 3 days ([[phab:T272247|T272247]])

=== 2021-01-15 ===
* 13:41 arturo: icinga downtime labstore1004 maintain-dbuser alert until 2021-01-19 ([[phab:T272125|T272125]])
* 09:47 arturo: labstore1004 maintain-dbusers affected by [[phab:T272127|T272127]] and [[phab:T272125|T272125]]
* 09:22 arturo: restart maintain-dbusers.service in labstore1004
* 08:19 dcaro: Merging the patch to disable write caches on ceph osds ([[phab:T271527|T271527]])

=== 2021-01-13 ===
* 17:03 arturo: remove cloudvirt1013 cloudvirt1032 cloudvirt1037 to the 'toobusy' host aggregate to prevent further CPU oversubscribing
* 12:40 arturo: try increasing systemd watchdog timeout for conntrackd in cloudnet1004 ([[phab:T268335|T268335]])
* 11:45 dcaro: https://gerrit.wikimedia.org/r/c/operations/puppet/+/654419 merged and deployed (and tested) ([[phab:T268877|T268877]])
* 11:40 dcaro: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/654419 that might affect the encapi service (puppet on cloud environment), no downtime expected though ([[phab:T268877|T268877]])
* 10:56 arturo: trying to cleanup dpkg package mess in cloudnet2002-dev
* 10:02 arturo: prevent floating IP allocation from neutron transport subnet: root@cloudcontrol1005:~# neutron subnet-update --allocation-pool start=185.15.56.244,end=185.15.56.244 cloud-instances-transport1-b-eqiad1 ([[phab:T271867|T271867]])

=== 2021-01-12 ===
* 10:33 arturo: reboot cloudnet1004
* 10:32 arturo: update firmware-bnx2x from 20190114-2 to 20200918-1~bpo10+1 on cloudnet1004 ([[phab:T271058|T271058]])

=== 2021-01-11 ===
* 10:22 arturo: doubling size of conntrack table in cloudnet servers https://gerrit.wikimedia.org/r/c/operations/puppet/+/655407 ([[phab:T271058|T271058]])
* 10:07 arturo: manually cleanup conntrack table in cloudnet1004 ([[phab:T271058|T271058]])
* 09:19 dcaro: cleaned up ~1800 snapshots, 109 remaining only, one for each host x image combination (plus some ephemeral ones while doing backups), closing the task ([[phab:T270478|T270478]])
* 08:39 dcaro: cleaning up dangling snapshots now that we have the new suffixed ones ([[phab:T270478|T270478]])

=== 2021-01-10 ===
* 16:02 andrewbogott: restarting rabbitmq-server on all eqiad1 cloudcontrols
* 15:54 andrewbogott: restating neutron-metadata-agent on cloudnet1004 due to many syslog complaints

=== 2021-01-08 ===
* 11:25 arturo: rebooting both cloudnet2002-dev/cloudnet2003-dev to make sure interfaces are set up correctl ([[phab:T271517|T271517]])
* 11:22 arturo: connecting cloudnet2002-dev cloudnet2003-dev back to vlan 2120 ([[phab:T271517|T271517]])
* 11:06 arturo: root@cloudcontrol2001-dev:~# openstack router set --external-gateway wan-transport-codfw --fixed-ip subnet=cloud-instances-transport1-b-codfw,ip-address=208.80.153.190 cloudinstances2b-gw ([[phab:T271517|T271517]])
* 11:02 arturo: root@cloudcontrol2001-dev:~# openstack router set --enable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw ([[phab:T271517|T271517]])
* 11:01 arturo: enabling neutron hacks in codfw1dev (cloudnet2002-dev, cloudnet2003-dev) ([[phab:T271517|T271517]])
* 10:55 arturo: aborrero@labtestvirt2003:~ $ sudo ifdown eno2.2107 ([[phab:T271517|T271517]])
* 10:55 arturo: aborrero@labtestvirt2003:~ $ sudo ifdown eno2.2120 ([[phab:T271517|T271517]])
* 10:53 arturo: root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 208.80.153.185 --ip-version 4 --network wan-transport-codfw --no-dhcp --subnet-range 208.80.153.184/29 cloud-instances-transport1-b-codfw ([[phab:T271517|T271517]])
* 10:40 dcaro: Finished tests, brining osd online (od.48) for eqiad ceph cluster ([[phab:T271417|T271417]])
* 09:59 dcaro: Started performance tests on sdc (od.48) for eqiad ceph cluster ([[phab:T271417|T271417]])
* 09:41 dcaro: Taking osd.48 from eqiad ceph cluster out to do performance tests ([[phab:T271417|T271417]])

=== 2021-01-07 ===
* 15:19 dcaro: Finished speed tests on cloudcephosd2001-dev, reprovisioning the osd.0 sdc ([[phab:T271417|T271417]])
* 14:39 dcaro: Starting speed tests on cloudcephosd2001-dev sdc ([[phab:T271417|T271417]])
* 12:54 dcaro: Taking osd.0 down on codfw ceph cluster to try the disk performance testing process ([[phab:T271417|T271417]])
* 11:35 arturo: merging dmz_cidr change ([[phab:T209082|T209082]], [[phab:T267779|T267779]])

=== 2021-01-05 ===
* 10:40 dcaro: removing dumps-[1..*] backups from cloudvirt1024 as they are not needed ([[phab:T271094|T271094]])

=== 2021-01-03 ===
* 07:06 dcaro: Got a network hiccup on cloudnet1004, keeping track here [[phab:T271058|T271058]]

=== 2020-12-28 ===
* 12:32 arturo: stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 ([[phab:T260692|T260692]])
* 12:32 arturo: stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 ([[phab:T260682|T260682]])
* 12:23 arturo: icinga downtime cloudvirt1026 disk space check until january 5 ([[phab:T260692|T260692]])
* 06:15 andrewbogott: restarting designate-central on cloudservices1003/1004.  I'm pretty sure they're distressed because of DB lag but it's worth a try

=== 2020-12-23 ===
* 15:38 andrewbogott: restarting rabbitmq on cloudcontrol1004; suspected leaks
* 15:33 andrewbogott: restarting each cloudcontrol galera node in turn to see if that quiets down the syncing warnings
* 12:08 arturo: move memory out of the swap in cloudcontrol1004 by disabling/enabling it (1Gb swap was being used)

=== 2020-12-22 ===
* 15:30 dcaro: cleaning up 6778 dangling snapshots for glance images in eqiad ([[phab:T270478|T270478]])
* 13:51 dcaro: merged patch to move wikidumpparse backups to cloudvirt1025 to free space on cloudvirt1026

=== 2020-12-19 ===
* 16:18 dcaro: gzipped a bunch of logs on cloudvirt1004 due to / being out of space
* 00:14 bstorm: truncated /var/log/debug.1 on cloudcontrol1003 which appears to be the exact same content as the user.log files anyway
* 00:10 bstorm: truncated /var/log/daemon.log.1 and the haproxy log
* 00:02 bstorm: truncated /var/log/messages.1 on cloudcontrol1003

=== 2020-12-18 ===
* 23:53 bstorm: truncated haproxy.log.1 on cloudcontrol1003
* 20:46 andrewbogott: setting  pg and pgp number to 4096 for eqiad1-compute as joachim thinks 8192 might be too much [[phab:T270305|T270305]]
* 17:09 dcaro: finished cleaning up the dangling snapshots from cloudvirt1026 ([[phab:T270478|T270478]])
* 17:08 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1026) ([[phab:T270478|T270478]])
* 17:06 dcaro: finished cleaning up the dangling snapshots from cloudvirt1025 ([[phab:T270478|T270478]])
* 17:05 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1025) ([[phab:T270478|T270478]])
* 17:00 dcaro: finished cleaning up the dangling snapshots from cloudvirt1021 ([[phab:T270478|T270478]])
* 16:58 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1021) ([[phab:T270478|T270478]])
* 16:56 dcaro: finished cleaning up the dangling snapshots from cloudvirt1022 ([[phab:T270478|T270478]])
* 16:55 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1022) ([[phab:T270478|T270478]])
* 16:54 dcaro: finished cleaning up the dangling snapshots from cloudvirt1023 ([[phab:T270478|T270478]])
* 16:51 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1023) ([[phab:T270478|T270478]])
* 16:47 dcaro: finished cleaning up the dangling snapshots from cloudvirt1024, freed ~12% of the capacity ([[phab:T270478|T270478]])
* 16:21 dcaro: removing dangling rbd snapshots (for backups on cloudvirt1024) ([[phab:T270478|T270478]])
* 16:13 andrewbogott: setting  autoscale to 'off' for both ceph pools (eqiad1-compute and eqiad1-glance-images) because we like how things are set and the autoscaler does not
* 10:33 dcaro: purging rbd snapshots for image fc6fb78b-4515-4dcc-8254-{{Gerrit|591b9fe01762}} ([[phab:T270478|T270478]])

=== 2020-12-17 ===
* 22:17 andrewbogott: correction to above, set the pg and pgp to 1024 for eqiad1-glance-images
* 22:16 andrewbogott: setting  pgp number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) [[phab:T270305|T270305]] (same as pg)
* 22:14 andrewbogott: setting  pg number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) [[phab:T270305|T270305]]
* 22:10 andrewbogott: setting  autoscale to 'warn' for both ceph pools (eqiad1-compute and eqiad1-glance-images)

=== 2020-12-16 ===
* 09:31 dcaro: removing invalid backups from cloudvirt1024 (196 in total) ([[phab:T269419|T269419]])

=== 2020-12-14 ===
* 17:42 dcaro: The removal freed ~12GB (still 100% usage :S) ([[phab:T269419|T269419]])
* 17:36 dcaro: removing invalid backups that have a valid copy ([[phab:T269419|T269419]])
* 15:43 dcaro: Merging the tagging for vm backups ([[phab:T267195|T267195]])
* 09:45 arturo: icinga downtime cloudvirt1024 for 6 days ([[phab:T269419|T269419]])

=== 2020-12-13 ===
* 09:11 _dcaro: running backup purge script on cloudvirt1024 ([[phab:T269419|T269419]])

=== 2020-12-10 ===
* 23:36 bstorm: cleaned up the logs for haproxy on cloudcontrol1003 by deleting all the gzipped ones and truncating the .1 file
* 11:56 dcaro: Freed some space on cloudvirt1024 by running the purge script ([[phab:T269419|T269419]])
* 09:17 dcaro: removing leaked dns record discordwiki.eqiad.wmflabs (clinic duty)

=== 2020-12-08 ===
* 18:01 dcaro: Host cloudvirt1030 up and running ([[phab:T216195|T216195]])
* 15:59 dcaro: Re-imaging host cloudvirt1030 ([[phab:T216195|T216195]])
* 14:18 dcaro: Host online cloudvirt1029 ([[phab:T216195|T216195]])
* 14:13 dcaro: Host re-imaged, doing tests cloudvirt1029 ([[phab:T216195|T216195]])
* 12:14 dcaro: Re-imaging cloudvirt1029 ([[phab:T216195|T216195]])

=== 2020-12-07 ===
* 18:33 andrewbogott: putting cloudvirt1023 back into service [[phab:T269467|T269467]]
* 15:55 andrewbogott: reimaging cloudvirt1028 for [[phab:T216195|T216195]]
* 14:49 dcaro: Re-imaging cloudvirt1027 ([[phab:T216195|T216195]])

=== 2020-12-05 ===
* 00:35 andrewbogott: moving cloudvirt1023 back into maintenance because [[phab:T269467|T269467]] continues to puzzle

=== 2020-12-04 ===
* 22:33 andrewbogott: moving cloudvirt1023 back into the ceph aggregate; it doesn't need upgrades after all  [[phab:T269467|T269467]]
* 22:24 andrewbogott: moving cloudvirt1023 out of the ceph aggregate and into maintenance for [[phab:T269467|T269467]]
* 21:06 andrewbogott: putting cloudvirt1025 and 1026 back into service because I'm pretty sure they're fixed.  [[phab:T269313|T269313]]
* 12:12 arturo: manually running `wmcs-purge-backups` again on cloudvirt1024 ([[phab:T269419|T269419]])
* 11:25 arturo: icinga downtime cloudvirt1024 for 6 days, to avoid paging noises ([[phab:T269419|T269419]])
* 11:25 arturo: last log line referencing cloudvirt1024 is a mistake ([[phab:T269313|T269313]])
* 11:24 arturo: icinga downtime cloudvirt1024 for 6 days, to avoid paging noises ([[phab:T269313|T269313]])
* 10:28 arturo: manually running `wmcs-purge-backups` on cloudvirt1024 ([[phab:T269419|T269419]])
* 10:23 arturo: setting expiration to 2020-12-03 to the oldest backy snapshot of every VM in cloudvirt1024 ([[phab:T269419|T269419]])
* 09:54 arturo: icinga downtime cloudvirt1025 for 6 days ([[phab:T269313|T269313]])

=== 2020-12-03 ===
* 23:21 andrewbogott: removing all osds on cloudcephosd1004 for rebuild, [[phab:T268746|T268746]]
* 21:45 andrewbogott: removing all osds on cloudcephosd1005 for rebuild, [[phab:T268746|T268746]]
* 19:51 andrewbogott: removing all osds on cloudcephosd1006 for rebuild, [[phab:T268746|T268746]]
* 17:01 arturo: icinga downtime cloudvirt1025 for 48h to debug network issue [[phab:T269313|T269313]]
* 16:56 arturo: rebooting cloudvirt1025 to debug network issue [[phab:T269313|T269313]]
* 16:38 dcaro: Rimaging cloudvirt1026 ([[phab:T216195|T216195]])
* 13:24 andrewbogott: removing all osds on cloudcephosd1008 for rebuild, [[phab:T268746|T268746]]
* 02:55 andrewbogott: removing all osds on cloudcephosd1009 for rebuild, [[phab:T268746|T268746]]

=== 2020-12-02 ===
* 20:04 andrewbogott: removing all osds on cloudcephosd1010 for rebuild, [[phab:T268746|T268746]]
* 17:25 arturo: [15:51] failovering neutron virtual router in eqiad1 ([[phab:T268335|T268335]])
* 15:36 arturo: conntrackd is now up and running in cloudnet1003/1004 nodes ([[phab:T268335|T268335]])
* 15:33 arturo: [codfw1dev] conntrackd is now up and running in cloudnet200x-dev nodes ([[phab:T268335|T268335]])
* 15:08 andrewbogott: removing all osds on cloudcephosd1012 for rebuild, [[phab:T268746|T268746]]
* 12:41 arturo: disable puppet in all cloudnet servers to merge conntrackd change [[phab:T268335|T268335]]
* 11:12 dcaro: Reset the properties for the flavor g2.cores8.ram16.disk1120 to correct quotes ([[phab:T269172|T269172]])
* 09:57 arturo: moved cloudvirts 1030, 1029, 1028, 1027, 1026, 1025 away from the 'standard' host aggregate to 'maintenance' ([[phab:T269172|T269172]])

=== 2020-12-01 ===
* 20:06 andrewbogott: removing all osds on cloudcephosd1014 for rebuild, [[phab:T268746|T268746]]
* 12:04 arturo: restarting neutron l3 agents to pick up config change
* 11:48 arturo: merging change to dmz_dir, detail list of private address https://gerrit.wikimedia.org/r/c/operations/puppet/+/641977

=== 2020-11-30 ===
* 18:12 andrewbogott: removing all osds from cloudcephosd1015 in order to investigate [[phab:T268746|T268746]]

=== 2020-11-29 ===
* 17:18 andrewbogott: cleaning up some logfiles in tools-sgecron-01 — drive is full

=== 2020-11-26 ===
* 22:58 andrewbogott: deleting /var/log/haproxy logs older than 7 days in cloudcontrol100x.  We need log rotation here it seems.
* 15:53 dcaro: Created private flavor g2.cores8.ram16.disk1120 for wikidumpparse ([[phab:T268190|T268190]])

=== 2020-11-25 ===
* 19:35 bstorm: repairing ceph pg `instructing pg 6.91 on osd.117 to repair`
* 09:31 _dcaro: The OSD seems to be up and running actually, though there's that misleading log, will leave it see if the cluster comes fully healthy ([[phab:T268722|T268722]])
* 08:54 _dcaro: Unsetting noup/nodown to allow re-shuffling of the pgs that osd.44 had, will try to rebuild it ([[phab:T268722|T268722]])
* 08:45 _dcaro: Tried resetting the class for osd.44 to ssd, no luck, the cluster is in noout/norebalance to avoid data shuffling (opened [[phab:T268722|T268722]])
* 08:45 _dcaro: Tried resetting the class for osd.44 to ssd, no luck, the cluster is in noout/norebalance to avoid data shuffling (opened root@cloudcephosd1005:/var/lib/ceph/osd/ceph-44# ceph osd crush set-device-class ssd osd.44)
* 08:19 _dcaro: Restarting serivce osd.44 resulted on osd.44 being unable to start due to some config inconsistency (can not reset class to hdd)
* 08:16 _dcaro: After enabling auto pg scaling on ceph eqiad cluster, osd.44 (cloudcephosd1005) got stuck, trying to restart the osd service
* 08:16 _dcaro: After enabling auto pg scaling on ceph eqiad cluster, osd.44 (cloudcephosd1005) got stuck, trying to restart

=== 2020-11-22 ===
* 17:40 andrewbogott: apt-get upgrade on cloudservices1003/1004
* 17:32 andrewbogott: upgrading Designate on cloudservices1003/1004 to Stein

=== 2020-11-20 ===
* 12:44 arturo: [codfw1dev] install conntrackd in cloudnet2003-dev/cloudnet2002-dev to research l3 agent HA reliability
* 09:26 arturo: incinga downtime labstore1006 RAID checks for 10 days ([[phab:T268281|T268281]])

=== 2020-11-17 ===
* 19:21 andrewbogott: draining cloudvirt1012 to experiment with libvirt/cpu things

=== 2020-11-15 ===
* 11:21 arturo: icinga downtime cloudbackup2002 for 48h ([[phab:T267865|T267865]])

=== 2020-11-10 ===
* 16:38 arturo: icinga downtime toolschecker for 2h becasue toolsdb maintenance ([[phab:T266587|T266587]])
* 11:24 arturo: [codfw1dev] enable puppet in puppetmaster01.cloudinfra-codfw1dev (disabled for unspecified reasons)

=== 2020-11-09 ===
* 12:42 arturo: restarted neutron l3 agent in cloudnet1003 bc it still had the old default route ([[phab:T265288|T265288]])
* 12:41 arturo: `root@cloudcontrol1005:~# neutron subnet-delete dcbb0f98-5e9d-4a93-8dfc-4e3ec3c44dcc` ([[phab:T265288|T265288]])
* 12:41 arturo: `root@cloudcontrol1005:~# neutron router-gateway-set  --fixed-ip subnet_id=7c6bcc12-212f-44c2-9954-{{Gerrit|5c55002ee371}},ip_address=185.15.56.244 cloudinstances2b-gw wan-transport-eqiad` ([[phab:T265288|T265288]])
* 12:19 arturo: subnet 185.1.5.56.240/29 has id 7c6bcc12-212f-44c2-9954-{{Gerrit|5c55002ee371}} in neutron ([[phab:T265288|T265288]])
* 12:19 arturo: `root@cloudcontrol1005:~# neutron subnet-create --gateway 185.15.56.241 --name cloud-instances-transport1-b-eqiad1 --ip-version 4 --disable-dhcp wan-transport-eqiad 185.15.56.240/29` ([[phab:T265288|T265288]])
* 12:15 arturo: icinga-downtime toolschecker for 2h ([[phab:T265288|T265288]])

=== 2020-11-02 ===
* 13:36 arturo: (typo: dcaro)
* 13:35 arturo: added dcar as projectadmin &amp; user ([[phab:T266068|T266068]])

=== 2020-10-29 ===
* 16:57 bstorm: silenced deployment-prep project alerts for 60 days since the downtime expired
* 08:12 arturo: force-powercycling cloudcephosd1006

=== 2020-10-25 ===
* 16:20 andrewbogott: adding cloudvirt1038 to the 'ceph' aggregate and removing from the 'spare' aggregate.  We need this space while waiting on network upgrades for empty cloudvirts ([[phab:T216195|T216195]])

=== 2020-10-23 ===
* 11:30 arturo: [codfw1dev] openstack --os-project-id cloudinfra-codfw1dev recordset create --type PTR --record nat.cloudgw.codfw1dev.wikimediacloud.org. --description &quot;created by hand&quot; 0-29.57.15.185.in-addr.arpa. 1.0-29.57.15.185.in-addr.arpa. ([[phab:T261724|T261724]])
* 10:09 arturo: [codf1dev] doing DNS changes for the cloudgw PoC, including designate and https://gerrit.wikimedia.org/r/c/operations/dns/+/635965 ([[phab:T261724|T261724]])

=== 2020-10-22 ===
* 10:46 arturo: [codfw1dev] rebooting cloudinfra-internal-puppetmaster-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud to try fixing some DNS weirdness
* 09:43 arturo: enabling puppet in cloucontrol1003 (message said &quot;please re-enable after 2020-10-22 06:00UTC&quot;)

=== 2020-10-21 ===
* 14:36 andrewbogott: running apt-get update &amp;&amp; apt-get install -y facter on all cloud-vps instances
* 10:31 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])
* 08:56 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])

=== 2020-10-20 ===
* 15:47 arturo: changing DNS recursor ACLs (https://gerrit.wikimedia.org/r/c/operations/puppet/+/635314) this can be reverted any time if it causes problems ([[phab:T261724|T261724]])
* 14:49 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code ([[phab:T261724|T261724]])

=== 2020-10-19 ===
* 01:41 andrewbogott: deleting all Precise base images
* 01:36 andrewbogott: deleting all unused Jessie base images

=== 2020-10-18 ===
* 23:26 andrewbogott: deleting all Trusty base images
* 21:50 andrewbogott: migrating all currently used ceph images to rbd

=== 2020-10-16 ===
* 09:29 arturo: [codfw1dev] still some DNS weirdness, investigating
* 09:25 arturo: [codfw1dev] hard-rebooting bastion-codfw1dev-02, seems in bad shape, doesn't even wake up in the virsh console
* 09:18 arturo: [codfw1dev] live-hacked cloudservices2002-dev /etc/powerdns/recursor.conf file to include cloud-codfw1dev-floating CIDR (185.15.57.0/29) while https://gerrit.wikimedia.org/r/c/operations/puppet/+/634050 is in review, so VMs with a floating IP can query the DNS recursor ([[phab:T261724|T261724]])
* 09:01 arturo: [codfw1dev] basic network connectivity seems stable after cleaning up everything related to address scopes ([[phab:T261724|T261724]])

=== 2020-10-15 ===
* 15:17 arturo: [codfw1dev] try cleaning up anything related to address scopes in the neutron database ([[phab:T261724|T261724]])
* 13:56 arturo: [codfw1dev] drop neutron l3 agent hacks in cloudnet2002/2003-dev ([[phab:T261724|T261724]])

=== 2020-10-13 ===
* 17:54 andrewbogott: rebuilding cloudvirt1021 for backy support
* 15:22 andrewbogott: draining cloudvirt1021 so I can rebuild it with backy support
* 14:19 andrewbogott: rebuilding cloudvirt1022 with backy support
* 14:03 andrewbogott: draining cloudvirt1022 so I can rebuild it with backy support
* 11:19 arturo: [codfw1dev] rebooting labtestvirt2003

=== 2020-10-09 ===
* 10:15 arturo: [codfwd1ev] root@cloudcontrol2001-dev:~# openstack router set --disable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw ([[phab:T261724|T261724]])
* 09:22 arturo: [codfwd1dev] rebooting cloudnet boxes for bridge and vlan changes ([[phab:T261724|T261724]])
* 09:12 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet delete 31214392-9ca5-4256-bff5-{{Gerrit|1e19a35661de}} (cloud-instances-transport1-b-codfw - 208.80.153.184/29) ([[phab:T261724|T261724]])
* 09:10 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack router set --external-gateway wan-transport-codfw --fixed-ip subnet=cloud-gw-transport-codfw,ip-address=185.15.57.10 cloudinstances2b-gw ([[phab:T261724|T261724]])
* 08:49 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.9 --no-dhcp --subnet-range 185.15.57.8/30 cloud-gw-transport-codfw ([[phab:T261724|T261724]])
* 08:47 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# openstack subnet delete a5ab5362-4ffb-4059-9ff7-{{Gerrit|391e22dcf3bc}} ([[phab:T261724|T261724]])

=== 2020-10-08 ===
* 16:17 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.8 --no-dhcp --subnet-range 185.15.57.8/31 cloud-gw-transport-codfw` (with a hack -- see task) ([[phab:T263622|T263622]])
* 16:03 arturo: [codfw1dev] briefly live-hacked python3-neutron source code in all 3 cloudcontrol2xxx-dev servers to workaround /31 network definition issue ([[phab:T263622|T263622]])
* 10:28 arturo: [codfw1dev] reimaging labtestvirt2003 (cloudgw) [[phab:T261724|T261724]]

=== 2020-10-06 ===
* 21:30 andrewbogott: moved cloudvirt1013 out of the 'ceph' aggregate and into the 'maintenance' aggregate for [[phab:T243414|T243414]]
* 21:29 andrewbogott: draining cloudvirt1013 for upgrade to 10G networking
* 14:45 arturo: icinga downtime every cloud* lab* host for 60 minutes for keystone maintenance

=== 2020-10-05 ===
* 17:40 bd808: `service uwsgi-labspuppetbackend restart` on cloud-puppetmaster-03 ([[phab:T264649|T264649]])

=== 2020-10-02 ===
* 11:05 arturo: [codfw1dev] restarting rabbitmq-server in all 3 control nodes, the l3 agent was misbehaving
* 09:16 arturo: [codfw1dev] trying the labtestvirt2003 (cloudgw) reimage again ([[phab:T261724|T261724]])

=== 2020-10-01 ===
* 16:06 arturo: rebooting cloudvirt1024 to validate changes to /etc/network/interfaces file
* 15:36 arturo: [codfw1dev] reimaging labtestvirt2003

=== 2020-09-30 ===
* 16:47 andrewbogott: rebooting cloudvir1032, 1033, 1034 for [[phab:T262979|T262979]]
* 13:28 arturo: enable puppet, reboot and pool back cloudvirt1031
* 13:27 arturo: extend icinga downtimes for another 120 mins
* 13:15 arturo: `aborrero@cloudcontrol1003:~$ sudo nova-manage placement sync_aggregates` after reading a hint in nova-api.log
* 13:02 arturo: rebooting cloudvirt1016 and moving it to the ceph host aggregate
* 12:55 arturo: rebooting cloudvirt1014 and moving it to the ceph host aggregate
* 12:51 arturo: rebooting cloudvirt1013 and moving it to the ceph host aggregate
* 12:39 arturo: root@cloudcontrol1005:~# openstack aggregate add host maintenance cloudvirt1031
* 12:36 arturo: rebooted cloudnet1003 (active) a couple of minutes ago
* 12:36 arturo: move cloudvirt1012 and cloudvirt1039 to the ceph aggregate
* 11:49 arturo: rebooting cloudvirt1039
* 11:46 arturo: rebooting cloudvirt1012
* 11:40 arturo: rebooting cloudnet1004 (standby) to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167 ([[phab:T262979|T262979]])
* 11:38 arturo: [codfw1dev] rebooting cloudnet2002-dev to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167
* 11:36 arturo: [codfw1dev] rebooting cloudnet2003-dev to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167
* 11:33 arturo: disabling puppet and downtiming every virt/net server in the fleet in preparation for merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/631167 ([[phab:T262979|T262979]])
* 09:32 arturo: rebooting cloudvirt1012 to investigate linuxbridge agent issues

=== 2020-09-29 ===
* 15:40 arturo: downgrade linux kernel from linux-image-4.19.0-11-amd64 to linux-image-4.19.0-10-amd64 on cloudvirt1012
* 14:47 arturo: rebooting cloudvirt1012, chasing config weirdness in the linuxbridge agent
* 14:05 andrewbogott: reimaging 1014 over and over in an attempt to get partman right
* 13:51 arturo: rebooting cloudvirt1012

=== 2020-09-28 ===
* 14:55 arturo: [jbond42] upgraded facter to v3 across the VM fleet
* 13:54 andrewbogott: moving cloudvirt1035 from aggregate 'spare' to 'ceph'.  We're going to need all the capacity we can get while converting older cloudvirts to ceph

=== 2020-09-24 ===
* 15:47 arturo: stopping/restarting rabbitmq-server in all cloudcontrol servers
* 15:45 arturo: restarting rabbitmq-server in cloudcontrol103
* 15:15 arturo: restarting floating_ip_ptr_records_updater.service in all 3 cloudcontrol servers to reset state after a DNS failure

=== 2020-09-18 ===
* 10:16 arturo: cloudvirt1039 libvirtd service issues were fixed with a reboot
* 09:56 arturo: rebooting cloudvirt1039 (spare) to try to fix some weird libvirtd failure
* 09:50 arturo: enabling puppet in cloudvirts and effectively merging patches from [[phab:T262979|T262979]]
* 08:59 arturo: disable puppet in all buster cloudvirts (cloudvirt[1024,1031-1039].eqiad.wmnet) to merge a patch for [[phab:T263205|T263205]] and [[phab:T262979|T262979]]
* 08:50 arturo: installing iptables from buster-bpo in cloudvirt1036 ([[phab:T263205|T263205]] and [[phab:T262979|T262979]])

=== 2020-09-15 ===
* 20:32 andrewbogott: rebooting cloudvirt1038 to see if it resolves [[phab:T262979|T262979]]
* 13:58 andrewbogott: draining cloudvirt1002 with wmcs-ceph-migrate

=== 2020-09-14 ===
* 14:21 andrewbogott: draining cloudvirt1001, migrating all VMs with wmcs-ceph-migrate
* 10:41 arturo: [codfw1dev] trying to get the bonding working for labtestvirt2003 ([[phab:T261724|T261724]])
* 09:47 arturo: installed qemu security update in eqiad1 cloudvirts ([[phab:T262386|T262386]])
* 09:43 arturo: [codfw1dev] installed qemu security update in codfw1dev cloudvirts ([[phab:T262386|T262386]])

=== 2020-09-09 ===
* 18:13 andrewbogott: restarting ceph-mon@cloudcephmon1003 in hopes that the slow ops reported are phantoms
* 18:01 andrewbogott: restarting ceph-mgr@cloudcephmon1003 in hopes that the slow ops reported are phantoms (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/EOWNO3MDYRUZKAK6RMQBQ5WBPQNLHOPV/)
* 17:40 andrewbogott: giving ceph pg autoscale another chance: ceph osd pool set eqiad1-compute pg_autoscale_mode on
* 00:05 bd808: Running wmcs-novastats-dnsleaks ([[phab:T262359|T262359]])

=== 2020-09-08 ===
* 21:48 bd808: Renamed FQDN prefixes to wikimedia.cloud scheme in cloudinfra-db01's labspuppet db ([[phab:T260614|T260614]])
* 14:29 andrewbogott: restarting nova-compute on all cloudvirts (everyone is upset from the reset switch failure)
* 14:18 arturo: restarting nova-fullstack service in cloudcontrol1003
* 14:17 andrewbogott: stopping apache2 on labweb1001 to make sure the Horizon outage is total

=== 2020-09-03 ===
* 09:31 arturo: icinga downtime cloud* servers for 30 mins ([[phab:T261866|T261866]])

=== 2020-09-02 ===
* 08:46 arturo: [codfw1dev] reimaging spare server labtestvirt2003 as debian buster ([[phab:T261724|T261724]])

=== 2020-09-01 ===
* 18:18 andrewbogott: adding drives on cloudcephosd100[3-5] to ceph osd pool
* 13:40 andrewbogott: adding drives on cloudcephosd101[0-2] to ceph osd pool
* 13:35 andrewbogott: adding drives on cloudcephosd100[1-3] to ceph osd pool
* 11:27 arturo: [codfw1dev] rebooting again cloudnet2002-dev after some network tests, to reset initial state ([[phab:T261724|T261724]])
* 11:09 arturo: [codfw1dev] rebooting cloudnet2002-dev after some network tests, to reset initial state ([[phab:T261724|T261724]])
* 10:49 arturo: disable puppet in cloudnet servers to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/623569/

=== 2020-08-31 ===
* 23:26 bd808: Removed stale lockfile at cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs:/var/lib/puppet/volatile/GeoIP/.geoipupdate.lock
* 11:20 arturo: [codfw1dev] livehacking https://gerrit.wikimedia.org/r/c/operations/puppet/+/615161 in the puppetmasters for tests before merging

=== 2020-08-28 ===
* 20:12 bd808: Running `wmcs-novastats-dnsleaks --delete` from cloudcontrol1003

=== 2020-08-26 ===
* 17:12 bstorm: Running 'ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf &quot;%k KB %p\n&quot; &gt; tools_large_files_20200826.txt' on labstore1004 [[phab:T261336|T261336]]

=== 2020-08-21 ===
* 21:34 andrewbogott: restarting nova-compute on cloudvirt1033; it seems stuck

=== 2020-08-19 ===
* 14:21 andrewbogott: rebooting cloudweb2001-dev, labweb1001, labweb1002 to address mediawiki-induced memleak

=== 2020-08-06 ===
* 21:02 andrewbogott: removing cloudvirt1004/1006 from nova's list of hypervisors; rebuilding them to  use as backup test hosts
* 20:06 bstorm: manually stopped the RAID check on cloudcontrol1003 [[phab:T259760|T259760]]

=== 2020-08-04 ===
* 18:54 bstorm: restarting mariadb on cloudcontrol1004 to setup parallel replication

=== 2020-08-03 ===
* 17:02 bstorm: increased db connection limit to 800 across galera cluster because we were clearly hovering at limit

=== 2020-07-31 ===
* 19:28 bd808: wmcs-novastats-dnsleaks --delete (lots of leaked fullstack-monitoring records to clean up)

=== 2020-07-27 ===
* 22:17 andrewbogott: ceph osd pool set compute pg_num 2048
* 22:14 andrewbogott: ceph osd pool set compute pg_autoscale_mode off

=== 2020-07-24 ===
* 19:15 andrewbogott: ceph mgr module enable pg_autoscaler
* 19:15 andrewbogott: ceph osd pool set compute pg_autoscale_mode on

=== 2020-07-22 ===
* 08:55 jbond42: [codfw1dev] upgrading hiera to version5
* 08:48 arturo: [codfw1dev] add jbond as user in the bastion-codfw1dev and cloudinfra-codfw1dev projects
* 08:45 arturo: [codfw1dev] enabled account creation in labtestwiki briefly for jbond42 to create an account

=== 2020-07-16 ===
* 10:48 arturo: merging change to neutron dmz_cidr https://gerrit.wikimedia.org/r/c/operations/puppet/+/613123 ([[phab:T257534|T257534]])

=== 2020-07-15 ===
* 23:15 bd808: Removed Merlijn van Deen from toollabs-trusted Gerrit group ([[phab:T255697|T255697]])
* 11:48 arturo: [codfw1dev] created DNS records (A and PTR) for bastion.bastioninfra-codfw1dev.codfw1dev.wmcloud.org &lt;-&gt; 185.15.57.2
* 11:41 arturo: [codfw1dev] add myself as projectadmin to the `bastioninfra-codfw1dev` project
* 11:39 arturo: [codfw1dev] created DNS zone `bastioninfra-codfw1dev.codfw1dev.wmcloud.org.` in the cloudinfra-codfw1dev project and then transfer ownership to the bastioninfra-codfw1dev project

=== 2020-07-14 ===
* 15:19 arturo: briefly set root@cloudnet1003:~ # sysctl net.ipv4.conf.all.accept_local=1 (in neutron qrouter netns) ([[phab:T257534|T257534]])
* 10:43 arturo: icinga downtime cloudnet* hosts for 30 mins to introduce new check https://gerrit.wikimedia.org/r/c/operations/puppet/+/612390 ([[phab:T257552|T257552]])
* 04:01 andrewbogott: added a wildcard *.wmflabs.org domain pointing at the domain proxy in project-proxy
* 04:00 andrewbogott: shortened the ttl on .wmflabs.org. to 300

=== 2020-07-13 ===
* 16:17 arturo: icinga downtime cloudcontrol[1003-1005].wikimedia.org for 1h for galera database movements

=== 2020-07-12 ===
* 17:39 andrewbogott: switched eqiad1 keystone from m5 to cloudcontrol galera

=== 2020-07-10 ===
* 20:26 andrewbogott: disabling nova api to move database to galera

=== 2020-07-09 ===
* 11:23 arturo: [codfw1dev] rebooting cloudnet2003-dev again for testing sysct/puppet behavior ([[phab:T257552|T257552]])
* 11:11 arturo: [codfw1dev] rebooting cloudnet2003-dev for testing sysct/puppet behavior ([[phab:T257552|T257552]])
* 09:16 arturo: manually increasing sysctl value of net.nf_conntrack_max in cloudnet servers ([[phab:T257552|T257552]])

=== 2020-07-06 ===
* 15:16 arturo: installing 'aptitude' in all cloudvirts

=== 2020-07-03 ===
* 12:51 arturo: [codfw1dev] galera cluster should be up and running, openstack happy ([[phab:T256283|T256283]])
* 11:44 arturo: [codfw1dev] restoring glance database backup from bacula into cloudcontrol2001-dev ([[phab:T256283|T256283]])
* 11:39 arturo: [codfw1dev] stopped mysql database in the galera cluster [[phab:T256283|T256283]]
* 11:36 arturo: [codfw1dev] dropped glance database in the galera cluster [[phab:T256283|T256283]]

=== 2020-07-02 ===
* 15:41 arturo: `sudo wmcs-openstack --os-compute-api-version 2.55 flavor create --private --vcpus 8 --disk 300 --ram 16384 --property aggregate_instance_extra_specs:ceph=true --description &quot;for packaging envoy&quot; bigdisk-ceph` ([[phab:T256983|T256983]])

=== 2020-06-29 ===
* 14:24 arturo: starting rabbitmq-server in all 3 cloudcontrol servers
* 14:23 arturo: stopping rabbitmq-server in all 3 cloudcontrol servers

=== 2020-06-18 ===
* 20:38 andrewbogott: rebooting cloudservices2003-dev due to a mysterious 'host down' alert on a secondary ip

=== 2020-06-16 ===
* 15:38 arturo: created by hand neutron port 9c0a9a13-e409-49de-9ba3-{{Gerrit|bc8ec4801dbf}} `paws-haproxy-vip` ([[phab:T295217|T295217]])

=== 2020-06-12 ===
* 13:23 arturo: DNS zone `paws.wmcloud.org` transferred to the PAWS project ([[phab:T195217|T195217]])
* 13:20 arturo: created DNS zone `paws.wmcloud.org` ([[phab:T195217|T195217]])

=== 2020-06-11 ===
* 19:19 bstorm_: proceeding with failback to labstore1004 now that DRBD devices are consistent [[phab:T224582|T224582]]
* 17:22 bstorm_: delaying failback labstore1004 for drive syncs [[phab:T224582|T224582]]
* 17:17 bstorm_: failing NFS back to labstore1004 to complete the upgrade process [[phab:T224582|T224582]]
* 16:15 bstorm_: failing over NFS for labstore1004 to labstore1005 [[phab:T224582|T224582]]

=== 2020-06-10 ===
* 16:09 andrewbogott: deleting all old cloud-ns0.wikimedia.org and cloud-ns1.wikimedia.org ns records in designate database [[phab:T254496|T254496]]

=== 2020-06-09 ===
* 15:25 arturo: icinga downtime everything cloud* lab* for 2h more ([[phab:T253780|T253780]])
* 14:09 andrewbogott: stopping puppet, all designate services and all pdns services on cloudservices1004 for [[phab:T253780|T253780]]
* 14:01 arturo: icinga downtime everything cloud* lab* for 2h ([[phab:T253780|T253780]])

=== 2020-06-05 ===
* 15:08 andrewbogott: trying to re-enable puppet without losing cumin contact, as per https://phabricator.wikimedia.org/T254589

=== 2020-06-04 ===
* 14:24 andrewbogott: disabling puppet on all instances for /labs/private recovery
* 14:23 arturo: disabling puppet on all instances for /labs/private recovery

=== 2020-05-28 ===
* 23:02 bd808: `/usr/local/sbin/maintain-dbusers --debug harvest-replicas` ([[phab:T253930|T253930]])
* 13:36 andrewbogott: rebuilding cloudservices2002-dev with Buster
* 00:33 andrewbogott: shutting down cloudservices2002-dev to see if we can live without it.  This is in anticipation or rebuilding it entirely for [[phab:T253780|T253780]]

=== 2020-05-27 ===
* 23:29 andrewbogott: disabling the backup job on cloudbackup2001 (just like last week) so the backup doesn't start while Brooke is rebuilding labstore1004 tomorrow.
* 06:03 bd808: `systemctl start mariadb` on clouddb1001 following reboot (take 2)
* 05:58 bd808: `systemctl start mariadb` on clouddb1001 following reboot
* 05:53 bd808: Hard reboot of clouddb1001 via Horizon. Console unresponsive.

=== 2020-05-25 ===
* 16:35 arturo: [codfw1dev] created zone `0-29.57.15.185.in-addr.arpa.` ([[phab:T247972|T247972]])

=== 2020-05-21 ===
* 19:23 andrewbogott: disabling puppet on cloudbackup2001 to prevent the backup job from starting during maintenance
* 19:16 andrewbogott: systemctl disable block_sync-tools-project.service on cloudbackup2001.codfw.wmnet to avoid stepping on current upgrade
* 15:48 andrewbogott: re-imaging cloudnet1003 with Buster

=== 2020-05-19 ===
* 22:59 bd808: `apt-get install mariadb-client` on cloudcontrol1003
* 21:12 bd808: Migrating wcdo.wcdo.eqiad.wmflabs to cloudvirt1023 ([[phab:T251065|T251065]])

=== 2020-05-18 ===
* 21:37 andrewbogott: rebuilding cloudnet2003-dev with Buster

=== 2020-05-15 ===
* 22:10 bd808: Added reedy as projectadmin in cloudinfra project ([[phab:T249774|T249774]])
* 22:05 bd808: Added reedy as projectadmin in admin project ([[phab:T249774|T249774]])
* 18:44 bstorm_: rebooting cloudvirt-wdqs1003 [[phab:T252831|T252831]]
* 15:47 bd808: Manually running wmcs-novastats-dnsleaks from cloudcontrol1003 ([[phab:T252889|T252889]])

=== 2020-05-14 ===
* 23:28 bstorm_: downtimed cloudvirt1004/6 and cloudvirt-wdqs1003 until tomorrow around this time [[phab:T252831|T252831]]
* 22:21 bstorm_: upgrading qemu-system-x86 on cloudvirt1006 to backports version [[phab:T252831|T252831]]
* 22:15 bstorm_: changing /etc/libvirt/qemu.conf and restarting libvirtd on cloudvirt1006 [[phab:T252831|T252831]]
* 21:12 andrewbogott: rebuilding cloudvirt1003-wdqs as part of [[phab:T252831|T252831]]
* 15:47 andrewbogott: moving cloudvirt1004 and cloudvirt1006 to the 'ceph' aggregate for [[phab:T252784|T252784]]
* 15:02 andrewbogott: moving all of cloudvirt100[1-9] into the 'toobusy' host aggregate.  These are slower, have spinning disks, and are due for replacement.

=== 2020-05-12 ===
* 20:33 andrewbogott: moving cloudvirt1023 to the 'standard' pool and out of the 'spare' pool
* 19:10 jeh: disable neutron-openvswitch-agent service on cloudvirt2001-dev.codfw [[phab:T248881|T248881]]
* 19:09 jeh: Shutdown the unused eno2 network interface on cloudvirt2001-dev.codfw to clear up monitoring errors [[phab:T248425|T248425]]
* 18:20 andrewbogott: moving cloudvirt1024 out of the 'maintenance' aggregate and into 'spare'
* 16:45 andrewbogott: restarting neutron-l3-agent on cloudnet1004 so it knows about all three cloudcontrols.  Leaving cloudnet1003 since restarting it there will cause network interruptions
* 14:06 arturo: icinga downtime everything for 2h for Debian Buster migration in some cloud components

=== 2020-05-09 ===
* 16:53 andrewbogott: rebuilding cloudcontrol2001-dev and 2003-dev with buster for [[phab:T252121|T252121]]

=== 2020-05-08 ===
* 19:02 bstorm_: moving tools-k8s-haproxy-2 from cloudvirt1021 to cloudvirt1017 to improve spread

=== 2020-05-05 ===
* 13:58 andrewbogott: rebuilding cloudcontrol2004-dev to test new puppet changes

=== 2020-05-04 ===
* 09:04 arturo: [codfw1dev] manually modify iptables ruleset to only allow SSH from WMF bastions on cloudservices2003-dev and cloudcontrol2004-dev ([[phab:T251604|T251604]])

=== 2020-04-21 ===
* 22:12 andrewbogott: moving cloudvirt1004 out of the 'standard' aggregate and into the 'maintenance' aggregate
* 16:01 jeh: restart cloudceph mon and osd services for openssl upgrades

=== 2020-04-15 ===
* 18:44 jeh: create indexes and views for grwikimedia [[phab:T245912|T245912]]

=== 2020-04-13 ===
* 15:07 jeh: restart memcached on labwebs to increase cache size [[phab:T145703|T145703]]

=== 2020-04-09 ===
* 19:57 andrewbogott: upgrading eqiad1 designate to rocky
* 16:52 andrewbogott: cleaned up a bunch of leaked .eqiad.wmflabs dns records

=== 2020-04-08 ===
* 19:20 andrewbogott: rotated password and api token for pdns servers on cloudservices1003 and cloudservices1004
* 14:54 arturo: `root@cloudcontrol1003:~# cp /etc/inputrc .inputrc` to solve some bash shortcut weirdness

=== 2020-04-07 ===
* 20:57 andrewbogott: service sssd stop; rm -rf /var/lib/sss/db*; service sssd start on tools-sgebastion-08

=== 2020-04-06 ===
* 22:39 andrewbogott: deleting bogus groups cn=b'project-bastion',ou=groups,dc=wikimedia,dc=org and cn=b'project-tools',ou=groups,dc=wikimedia,dc=org from ldap
* 17:42 arturo: [codfw1dev] transferred DNS zone 57.15.185.in-addr.arpa. to the cloudinfra-codfw1dev project ([[phab:T247972|T247972]])
* 17:39 arturo: [codfw1dev] `openstack zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description &quot;floating IPs subnet&quot; 57.15.185.in-addr.arpa.` ([[phab:T247972|T247972]])
* 16:23 arturo: restarting apache2 in cloudcontrol1003/1004 to pick up latest wmfkeystonehooks changes [[phab:T249494|T249494]]

=== 2020-04-02 ===
* 20:59 jeh: codfw1dev clear VM error states and start bastions, puppet master and database

=== 2020-04-01 ===
* 16:27 arturo: [codfw1dev] enable puppet across the fleet clean vxlan changes ([[phab:T248881|T248881]])

=== 2020-03-31 ===
* 12:35 arturo: [codfw1dev] restarting VMs: designaterockytest14, bastion-codfw1dev-0[1,2] ([[phab:T248881|T248881]])
* 12:34 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudvirt2001-dev ([[phab:T248881|T248881]])
* 12:25 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudnet200[2,3]-dev ([[phab:T248881|T248881]])
* 11:45 arturo: [codfw1dev] rebooting cloudvirt2003-dev to pick up latest kernel update. Otherwise modprobe is confused trying to load modules and openvswitch won't start ([[phab:T248881|T248881]])
* 10:40 arturo: [codfw1dev] installing neutron-openvswitch-agent on cloudvirt2003-dev ([[phab:T248881|T248881]])
* 10:09 arturo: [codfw1dev] reboot cloudnet2003-dev into linux 4.9 (was using 4.14 from a testing operation in 2020-03-10)

=== 2020-03-30 ===
* 23:42 bstorm_: deleted &quot;Kubernetes Cluster&quot; and &quot;Kubernetes Performance&quot; dashboards [[phab:T246689|T246689]]
* 16:44 arturo: [codfw1dev] installing package neutron-openvswitch-agent in cloudvirt2002-dev ([[phab:T248881|T248881]])
* 16:42 andrewbogott: restarting l3 agents on cloudnets in codfw1dev after applying https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/584188/

=== 2020-03-27 ===
* 21:28 bd808: Created huggle.wmcloud.org Designate zone and allocated it to the huggle project
* 19:51 jeh: start haproxy on cloudcontrol2003-dev.wikimedia.org

=== 2020-03-26 ===
* 15:01 arturo: icinga downtime cloudvirt* cloudcontrol* cloudnet* lab* cloudstore*
* 15:01 andrewbogott: beginning openstack upgrade window for [[phab:T242766|T242766]]
* 12:32 arturo: [codfw1dev] downgraded systemd, libsystemd0, udev and friends to the non-backports versions ([[phab:T247013|T247013]])

=== 2020-03-25 ===
* 19:29 andrewbogott: dumping a bunch of VMs on cloudvirt1015 to see if it still crashes
* 17:56 jeh: add labweb1002 back into the pool - completed horizon testing [[phab:T240852|T240852]]
* 17:09 jeh: depool labweb1002 for horizon testing [[phab:T240852|T240852]]

=== 2020-03-24 ===
* 19:41 jeh: switch cloudvirt1016 from maintenance to standard host aggregate [[phab:T243327|T243327]]
* 15:31 andrewbogott: restarting nova-conductor and nova-api on cloudcontrol1003 and cloudcontrol1004

=== 2020-03-23 ===
* 21:41 jeh: restart neutron-l3-agent on cloudnet100[3,4] to pickup policy.yaml changes
* 13:28 jeh: disable puppet on labweb100[1,2] to enable horizon event traces [[phab:T240852|T240852]]
* 10:26 arturo: restarting apache in both labweb1001/labweb1002 upon reports of returning 500s

=== 2020-03-21 ===
* 14:23 andrewbogott: restarting apache2 on labweb1001 and 1002

=== 2020-03-18 ===
* 19:17 andrewbogott: deleted a bunch of records from the pdns database on cloudservices1003/1004 which had a record name but the content (where an IP address should be) was NULL, e.g. m.wikidata.beta.wmflabs.org.
* 10:55 arturo: [codfw1dev] deleting BGP agent, undoing changes we did for [[phab:T245606|T245606]]

=== 2020-03-14 ===
* 17:40 jeh: restart maintain-dbusers on labstore1004 [[phab:T247654|T247654]]

=== 2020-03-13 ===
* 12:39 arturo: [codfw1dev] reintroduce address scopes for another round of testing [[phab:T244851|T244851]]
* 12:17 arturo: [codfw1dev] enabling puppet in cloudnet200x-dev servers after merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/579259 ([[phab:T247505|T247505]])

=== 2020-03-12 ===
* 22:29 bstorm_: running puppet across all dumps mounts to make sure active links are shifted to labstore1006

=== 2020-03-11 ===
* 18:38 jeh: set icingia downtime until 2020-03-23 on CODFW cloud[control,net,virt] hosts during openstack upgrades
* 12:50 arturo: [codfw1dev] several tests creating/deleting address scopes ([[phab:T244727|T244727]] [[phab:T247135|T247135]] [[phab:T246887|T246887]] [[phab:T245606|T245606]])
* 12:46 arturo: [codfw1dev] disable routing_source_ip in l3 agents for testing proposal detailed at https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Network_refresh#Eliminate_routing_source_ip_address ([[phab:T244727|T244727]])

=== 2020-03-10 ===
* 17:02 arturo: [codfw1dev] deleting address scopes, bad interaction with our custom NAT setup [[phab:T247135|T247135]]
* 13:55 arturo: [codfw1dev] rebooting cloudnet2003-dev into linux kernel 4.14 for testing stuff related to [[phab:T247135|T247135]]

=== 2020-03-09 ===
* 18:09 arturo: enabling puppet in cloudvirt1006, all services have been restored
* 17:59 arturo: deleted the neutron bridge on cloudvirt1006, for testing stuff related to the queens upgrade
* 17:58 arturo: stopped neutron-linuxbridge-agent and nova-compute in cloudvirt1006 for testing stuff related to the queens upgrade

=== 2020-03-06 ===
* 14:54 andrewbogott: draining all instances off of cloudvirt1006 for [[phab:T246908|T246908]]

=== 2020-03-05 ===
* 14:24 arturo: [codfw1dev] we just enabled BGP session between cloudnet2xxx-dev and cr1-codfw ([[phab:T245606|T245606]])
* 13:07 arturo: [codfw1dev] move the extra IP address for BGP in cloudnet200x-dev servers from eno2.2120 to the br-external bridge device ([[phab:T245606|T245606]])
* 13:06 arturo: [codfw1dev] upgrade neutron-dynamic-routing packages in cloudnet200X-dev and cloudcontrol200X-dev servers to 11.0.0-2~bpo9+1 ([[phab:T245606|T245606]])

=== 2020-03-04 ===
* 22:22 andrewbogott: upgrading designate on cloudservices1003/1004 to Queens
* 22:09 andrewbogott: moving cloudvirt1006 into the maintenance aggregate for [[phab:T246908|T246908]]
* 21:37 bd808: Running wmcs-wikireplica-dns to add service names for ngwikimedia.*.db.svc.eqiad.wmflabs ([[phab:T240772|T240772]])
* 21:14 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1009 ([[phab:T246056|T246056]])
* 21:11 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1010 ([[phab:T246056|T246056]])
* 21:08 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1011 ([[phab:T246056|T246056]])
* 21:05 bd808: Running `sudo maintain-meta_p --all-databases --purge` on labsdb1002 ([[phab:T246056|T246056]])

=== 2020-03-02 ===
* 16:54 arturo: [codfw1dev] deleted python3-os-ken debian package in cloudnet2003-dev which was installed by hand and had depedency issues

=== 2020-02-29 ===
* 16:32 bstorm_: downtimed the smart alert on cloudvirt1009 until Monday since apparently predictive failures flap [[phab:T244986|T244986]]

=== 2020-02-26 ===
* 22:03 jeh: powering down cloudvirt1014 for hardware maintenance

=== 2020-02-25 ===
* 16:08 andrewbogott: changing neutron's rabbitmq password because oslo is having trouble parsing some of the characters in the password
* 15:26 andrewbogott: updated the cell_mapping record in the nova_api database to add the second rabbitmq server  to the transport_url field
* 15:26 andrewbogott: updated the cell_mapping record in the nova_api database to set the db uri to 'mysql+pymysql' -- this in response to a deprecation notice

=== 2020-02-24 ===
* 12:16 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker cr2-codfw` ([[phab:T245606|T245606]])
* 12:16 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker cr1-codfw` ([[phab:T245606|T245606]])
* 12:09 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.187 --remote-as 65002 cr2-codfw` ([[phab:T245606|T245606]])
* 12:09 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.186 --remote-as 65002 cr1-codfw` ([[phab:T245606|T245606]])
* 12:06 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-delete 17b8c2a3-f0ce-4d50-a265-18ccac703c61` ([[phab:T245606|T245606]])
* 10:59 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker bgppeer` ([[phab:T245606|T245606]])
* 10:56 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-peer-create --peer-ip 208.80.153.185 --remote-as 65002 bgppeer` ([[phab:T245606|T245606]])

=== 2020-02-21 ===
* 12:48 arturo: [codfw1dev] running `root@cloudcontrol2001-dev:~# neutron bgp-speaker-network-add bgpspeaker wan-transport-codfw` ([[phab:T245606|T245606]])
* 12:46 arturo: [codfw1dev] created bgpspeaker for AS64711 ([[phab:T245606|T245606]])
* 12:42 arturo: [codfw1dev] run `sudo neutron-db-manage upgrade head` to upgrade the db schema for neutron bgp tables
* 11:51 arturo: [codfw1dev] create a neutron subnet pool per each subnet objects we have and manually update DB to inter-associate them ([[phab:T245606|T245606]])
* 11:49 arturo: [codfw1dev] rename neutron address scope `no-nat` to `bgp` ([[phab:T245606|T245606]])
* 11:37 arturo: [codfw1dev] cleanup unused neutron subnet pools from previous address scope tests ([[phab:T244851|T244851]])

=== 2020-02-20 ===
* 19:22 andrewbogott: updating designate pool config for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/572213/
* 15:33 andrewbogott: migrating all VMs on cloudvirt1014 to cloudvirt1022
* 13:35 arturo: [codfw1dev] disable puppet in cloudcontrol servers to hack neutron.conf for tests related to [[phab:T245606|T245606]]
* 13:33 arturo: [codfw1dev] disable puppet in cloudnet servers to hack neutron.conf for tests related to [[phab:T245606|T245606]]

=== 2020-02-18 ===
* 22:19 andrewbogott: transferred the tools.wmcloud.org. to the tools project
* 22:16 andrewbogott: moved wmcloud.org dns domain to the cloud-infra project
* 21:02 andrewbogott: adding .eqiad1.wikimedia.cloud records to all existing eqiad1 VMs, updating all eqiad1 internal pointer records to reference the new eqiad1.wikimedia.cloud fqdns.
* 09:44 arturo: deleted DNS zone wmcloud.org and try re-creating it

=== 2020-02-14 ===
* 10:35 arturo: running `root@cloudcontrol2001-dev:~# designate server-create --name ns1.openstack.codfw1dev.wikimediacloud.org.` ([[phab:T243766|T243766]])
* 10:32 arturo: running `root@cloudcontrol1004:~# designate server-create --name ns1.openstack.eqiad1.wikimediacloud.org.` ([[phab:T243766|T243766]])
* 10:32 arturo: running `root@cloudcontrol1004:~# designate server-create --name ns0.openstack.eqiad1.wikimediacloud.org.` ([[phab:T243766|T243766]])

=== 2020-02-12 ===
* 13:38 arturo: [codfw1dev] add reference to subnetpool to the instance subnet `MariaDB [neutron]&gt; update subnets set subnetpool_id='d129650d-d4be-4fe1-b13e-6edb5565cb4a' where id = '7adfcebe-b3d0-4315-92fe-e8365cc80668';` ([[phab:T244851|T244851]])

=== 2020-02-11 ===
* 13:46 arturo: [codfw1dev] creating some neutron objects to investigate [[phab:T244851|T244851]] (subnets, subnet pools, address scopes, ...)
* 12:40 arturo: [codfw1dev] delete unknown address scope 'wmcs-v4-scope': `root@cloudcontrol2001-dev:~# openstack address scope delete 078cfd71-117b-4aac-9197-6ebbbb7dd3de` ([[phab:T244851|T244851]])
* 12:40 arturo: [codfw1dev] delete unknown subnet pool 'cloudinstancesb-v4-pool0': `root@cloudcontrol2001-dev:~# openstack subnet pool delete d23a9b88-5c3d-4a53-ab88-053233a75365` ([[phab:T244851|T244851]])

=== 2020-02-07 ===
* 18:11 jeh: shutdown cloudvirt1016 for hardware maintenance [[phab:T241882|T241882]]

=== 2020-02-06 ===
* 14:44 jeh: update apt packages on cloudvirt1015 [[phab:T220853|T220853]]
* 14:28 jeh: run hardware tests on cloudvirt1015 [[phab:T220853|T220853]]

=== 2020-01-28 ===
* 17:24 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# designate server-create --name ns0.openstack.codfw1dev.wikimediacloud.org. ([[phab:T243766|T243766]])
* 10:18 arturo: [codfw1dev] created DNS record `bastion-codfw1dev-01.codfw1dev.wmcloud.org A 185.15.57.2` ([[phab:T242976|T242976]], [[phab:T229441|T229441]])
* 10:13 arturo: [codfw1dev] the zone `codfw1dev.wmcloud.org` belongs now to the `cloudinfra-codfw1dev` project ([[phab:T242976|T242976]])
* 10:11 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description &quot;main DNS domain for public addresses&quot; --email &quot;root@wmflabs.org&quot; --type PRIMARY --ttl 3600 codfw1dev.wmcloud.org.` ([[phab:T242976|T242976]] and [[phab:T243766|T243766]])
* 09:53 arturo: restart apache2 in labweb1001/1002 because horizon errors
* 09:47 arturo: created DNS zone wmcloud.org in eqiad1, transfer it to the cloudinfra project ([[phab:T242976|T242976]]) right now only use is to delegate codfw1dev.wmcloud.org subdomain to designate in the other deployment

=== 2020-01-27 ===
* 12:45 arturo: [codfw1dev] manually move the new domain to the `cloudinfra-codfw1dev` project clouddb2001-dev: `[designate]&gt; update zones set tenant_id='cloudinfra-codfw1dev' where id = '4c75410017904858a5839de93c9e8b3d';` [[phab:T243556|T243556]]
* 12:44 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description &quot;main DNS domain for VMs&quot; --email &quot;root@wmflabs.org&quot; --type PRIMARY --ttl 3600 codfw1dev.wikimedia.cloud.` [[phab:T243556|T243556]]

=== 2020-01-24 ===
* 15:10 jeh: remove icinga downtime for cloudvirt1013 [[phab:T241313|T241313]]
* 12:52 arturo: repooling cloudvirt1013 after HW got fixed ([[phab:T241313|T241313]])

=== 2020-01-21 ===
* 17:43 bstorm_: remounting /mnt/nfs/dumps-labstore1007.wikimedia.org/ on all dumps-mounting projects
* 10:24 arturo: running `sudo systemctl restart apache2.service` in both labweb servers to try mitigating [[phab:T240852|T240852]]

=== 2020-01-15 ===
* 16:59 bd808: Changed the config for cloud-announce mailing list so that lsit admins do not get bounce unsubscribe notices

=== 2020-01-14 ===
* 14:03 arturo: icinga downtime all cloudvirts for another 2h for fixing some icinga checks
* 12:04 arturo: icinga downtime toolchecker for 2 hours for openstack upgrades [[phab:T241347|T241347]]
* 12:02 arturo: icinga downtime cloud* labs* hosts for 2 hours for openstack upgrades [[phab:T241347|T241347]]
* 04:26 andrewbogott: upgrading designate on cloudservices1003/1004

=== 2020-01-13 ===
* 13:34 arturo: [¢odfw1dev] prevent neutron from allocating floating IPs from the wrong subnet by doing `neutron subnet-update --allocation-pool start=208.80.153.190,end=208.80.153.190 cloud-instances-transport1-b-codfw` ([[phab:T242594|T242594]])

=== 2020-01-10 ===
* 13:27 arturo: cloudvirt1009: virsh undefine i-000069b6. This is tools-elastic-01 which is running on cloudvirt1008 (so, leaked on cloudvirt1009)

=== 2020-01-09 ===
* 11:12 arturo: running `MariaDB [nova_eqiad1]&gt; update quota_usages set in_use='0' where project_id='etytree';` ([[phab:T242332|T242332]])
* 11:11 arturo: running `MariaDB [nova_eqiad1]&gt; select * from quota_usages where project_id = 'etytree';` ([[phab:T242332|T242332]])
* 10:32 arturo: ran `root@cloudcontrol1004:~# nova-manage project quota_usage_refresh --project etytree`

=== 2020-01-08 ===
* 10:53 arturo: icinga downtime all cloudvirts for 30 minutes to re-create all canary VMs&quot;

=== 2020-01-07 ===
* 11:12 arturo: icinga-downtime everything cloud* for 30 minutes to merge nova scheduler changes
* 10:02 arturo: icinga downtime cloudvirt1009 for 30 minutes to re-create canary VM ([[phab:T242078|T242078]])

=== 2020-01-06 ===
* 13:45 andrewbogott: restarting nova-api and nova-conductor on cloudcontrol1003 and 1004

=== 2020-01-04 ===
* 16:34 arturo: icinga downtime cloudvirt1024 for 2 months because hardware errors ([[phab:T241884|T241884]])

=== 2019-12-31 ===
* 11:46 andrewbogott: I couldn't!
* 11:40 andrewbogott: restarting cloudservices2002-dev to see if I can reproduce an issue I saw earlier

=== 2019-12-25 ===
* 10:13 arturo: icinga downtime for 30 minutes the whole cloud* lab* fleet to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/560575 (will restart some openstack components)

=== 2019-12-24 ===
* 15:13 arturo: icinga downtime all the lab* fleet for nova password change for 1h
* 14:39 arturo: icinga downtime all the cloud* fleet for nova password change for 1h

=== 2019-12-23 ===
* 11:13 arturo: enable puppet in cloudcontrol1003/1004
* 10:40 arturo: disable puppet in cloudcontrol1003/1004 while doing changes related to python-ldap

=== 2019-12-22 ===
* 23:48 andrewbogott: restarting nova-conductor and nova-api on cloudcontrol1003 and 1004
* 09:45 arturo: cloudvirt1013 is back (did it alone) [[phab:T241313|T241313]]
* 09:37 arturo: cloudvirt1013 is down for good. Apparently powered off. I can't even reach it via iLO

=== 2019-12-20 ===
* 12:43 arturo: icinga downtime cloudmetrics1001 for 128 hours

=== 2019-12-18 ===
* 12:55 arturo: [codfw1dev] created a new subnet neutron object to hold the new CIDR for floating IPs (cloud-codfw1dev-floating - 185.15.57.0/29) [[phab:T239347|T239347]]

=== 2019-12-17 ===
* 07:21 andrewbogott: deploying horizon/train to labweb1001/1002

=== 2019-12-12 ===
* 06:11 arturo: schedule 4h downtime for labstores
* 05:57 arturo: schedule 4h downtime for cloudvirts and other openstack components due to upgrade ops

=== 2019-12-02 ===
* 06:28 andrewbogott: running  nova-manage db sync on eqiad1
* 06:27 andrewbogott: running nova-manage cell_v2 map_cell0 on eqiad1

=== 2019-11-21 ===
* 16:07 jeh: created replica indexes and views for szywiki [[phab:T237373|T237373]]
* 15:48 jeh: creating replica indexes and views for shywiktionary [[phab:T238115|T238115]]
* 15:48 jeh: creating replica indexes and views for gcrwiki [[phab:T238114|T238114]]
* 15:46 jeh: creating replica indexes and views for minwiktionary [[phab:T238522|T238522]]
* 15:36 jeh: creating replica indexes and views for gewikimedia [[phab:T236404|T236404]]

=== 2019-11-18 ===
* 19:27 andrewbogott: repooling labsdb1011
* 18:54 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1011    [[phab:T238480|T238480]]
* 18:44 andrewbogott: depooling labsdb1011 and killing remaining user queries  [[phab:T238480|T238480]]
* 18:42 andrewbogott: repooled labsdb1009 and 1010  [[phab:T238480|T238480]]
* 18:19 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1010    [[phab:T238480|T238480]]
* 18:18 andrewbogott: depooling labsdb1010, killing remaining user queries
* 17:46 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1009    [[phab:T238480|T238480]]
* 17:38 andrewbogott: depooling labsdb1009, killing remaining user queries
* 16:54 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1012   [[phab:T237509|T237509]]

=== 2019-11-15 ===
* 20:04 andrewbogott: repool labdb1011 ([[phab:T237509|T237509]])
* 19:29 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1011
* 19:25 andrewbogott: depooling labsdb1011, killing remaining queries
* 19:25 andrewbogott: repooling labsdb1010
* 18:59 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1012
* 18:57 andrewbogott: running maintain-views --all-databases --replace-all —clean on labsdb1010
* 18:54 andrewbogott: depooling labsdb1010, killing remaining user queries
* 18:54 andrewbogott: depooled labsdb1009, ran maintain-views —clean —all-databases —replace-all, repooled

=== 2019-11-11 ===
* 13:10 arturo: cloudweb2001-dev: disable puppet and redirect stderr in the loadExitNodes.php cron script to prevent cronspam while we investigate the cause of the issue ([[phab:T237971|T237971]])

=== 2019-11-05 ===
* 11:59 arturo: icinga downtime for 1h cloudcontrol1004, cloudnet1003, cloudvirt1017/1020/1022 for PDU operations in the rack [[phab:T227542|T227542]]

=== 2019-11-04 ===
* 21:55 andrewbogott: deleting a ton of wikitech hiera pages that were either no-ops or refer to nonexistent VMs or prefixes

=== 2019-10-31 ===
* 11:01 arturo: icinga-downtimed cloudvirt1030 and cloudservices1003 for 1h due to PDU upgrade operations [[phab:T227543|T227543]]

=== 2019-10-30 ===
* 22:43 jeh: reboot cloud-bootstrapvz-stretch to resolve bad bootstrapvz build

=== 2019-10-29 ===
* 10:52 arturo: icinga downtime cloudvirt1001/1002/1024/1018/1012/1009/1015/1008 for 1h [[phab:T227538|T227538]]

=== 2019-10-25 ===
* 10:45 arturo: icinga downtime toolschecker for 1 to upgrade clouddb1002 mariadb (toolsdb secondary) ([[phab:T236384|T236384]] , [[phab:T236420|T236420]])

=== 2019-10-24 ===
* 12:30 arturo: starting cloudvirt1019, PDU operations ended ([[phab:T227540|T227540]])
* 11:58 arturo: icinga downtime for 2h ([[phab:T227540|T227540]]) cloudvirt1019
* 11:15 arturo: poweroff cloudvirt1019 during the PDU operations ([[phab:T227540|T227540]])
* 11:10 arturo: icinga downtime for 2h ([[phab:T227540|T227540]]) toolschecker
* 10:58 arturo: icinga downtime for 1h ([[phab:T227540|T227540]]) cloudvirt100[3-7], cloudvirt1019, cloudvirt1016, cloudvirt1021, cloudvirt1013, cloudnet1004

=== 2019-10-23 ===
* 09:23 arturo: cloudvirt1026 reboot ended OK
* 09:12 arturo: rebooting cloudvirt1026 for kernel upgrade
* 09:09 arturo: cloudvirt1025 reboot ended OK
* 09:00 arturo: rebooting cloudvirt1025 for kernel upgrade
* 08:51 arturo: icinga downtime cloudvirt1025/1026 for reboots

=== 2019-10-18 ===
* 16:01 arturo: created the `eqiad1.wikimedia.cloud` DNS zone ([[phab:T235846|T235846]])
* 14:27 andrewbogott: deleted a bunch of leaked VMS from earlier today from the admin-monitoring project.   Fullstack leaks due to an api outage, maybe?
* 10:44 arturo: double max_message_size from 40KB to 80KB in the cloud-admin mailing list. A simple email with a couple of quotes can go over the 40KB limit.

=== 2019-10-16 ===
* 21:59 jeh: resync wiki replica tool and user accounts [[phab:T235697|T235697]]
* 09:40 arturo: reboot of cloudvirt1030 went fine
* 09:28 arturo: reboot of cloudvirt1029 went fine
* 09:28 arturo: rebooting cloudvirt1030 for kernel updates
* 09:12 arturo: rebooting cloudvirt1029 for kernel updates
* 09:11 arturo: reboot of cloudvirt1028 went fine
* 09:00 arturo: rebooting cloudvirt1028 for kernel updates
* 08:56 arturo: icinga downtime cloudvirt[1028-1030].eqiad.wmnet for 1h for reboots

=== 2019-10-15 ===
* 13:30 jeh: creating indexes and views for banwiki [[phab:T234770|T234770]]

=== 2019-10-10 ===
* 18:55 bd808: Created indexes and views for nqowiki ([[phab:T230543|T230543]])
* 11:59 arturo: network switch hardware is down affecting cloudvirt1025/1026 ([[phab:T227536|T227536]]) VMs are supposed to be online but unreachable

=== 2019-10-09 ===
* 10:44 arturo: cloudvirt1013 rebooted well
* 10:32 arturo: cloudvirt1013 is rebooting
* 10:32 arturo: cloudvirt1012 rebooted just fine (very slow, 35 VMs)
* 10:21 arturo: cloudvirt1012 is rebooting
* 10:19 arturo: cloudvirt1009 rebooted just fine (very slow though)
* 10:07 arturo: cloudvirt1009 is rebooting
* 10:06 arturo: cloudvirt1008 rebooted just fine (very slow though)
* 09:58 arturo: cloudvirt1008 is rebooting
* 09:52 arturo: icinga downtime toolschecker, paws, etc for 2h, because cloudvirt reboots

=== 2019-10-07 ===
* 14:07 arturo: horizon is disabled for maintenance ([[phab:T212302|T212302]])
* 14:00 arturo: starting scheduled maintenance: upgrading eqiad1 from openstack mitaka to newton

=== 2019-10-02 ===
* 15:23 arturo: codfw1dev renaming net/subnet objects to a more modern naming scheme [[phab:T233665|T233665]]
* 12:49 arturo: codfw1dev delete all floating ip allocations in the deployment for mangling the network config for testing [[phab:T233665|T233665]]
* 12:47 arturo: codfw1dev deleting all VMs in the deployment for mangling the network config for testing [[phab:T233665|T233665]]
* 11:08 arturo: codfw1dev rebooting cloudnet2002-dev and cloudnet2003-dev for testing [[phab:T233665|T233665]]
* 10:31 arturo: codfw1dev: add cloudinstances2b-gw router to the l3 agent in cloudnet2003-dev
* 09:59 arturo: codfw1dev: cleanup leftover &quot;HA port tenant admin&quot; in neutron (ports from missing servers)
* 09:46 arturo: codfw1dev: cleanup leftover neutron agents

=== 2019-09-30 ===
* 10:21 arturo: we installed ferm in every VM by mistake. Deleting it and forcing a puppet agent run to try to go back to a clean state.
* 09:38 arturo: downtime toolschecker for 24h
* 09:33 arturo: force update ferm cloud-wide (in all VMs) for [[phab:T153468|T153468]]

=== 2019-08-18 ===
* 10:39 arturo: rebooting cloudvirt1023 for new interface names configuration
* 10:34 arturo: downtimed cloudvirt1023 for 2 days

=== 2019-08-05 ===
* 17:17 bd808: Set downtime on gridengine and kubernetes webservice checks in icinga until 2019-09-02 (flaky tests)

=== 2019-07-29 ===
* 20:14 bd808: Restarted maintain-kubeusers on tools-k8s-master-01 ([[phab:T194859|T194859]])

=== 2019-07-25 ===
* 12:32 arturo: eqiad1/glance: debian-9.9-stretch image deprecates debian-9.8-stretch ([[phab:T228983|T228983]])
* 09:59 arturo: (codfw1dev) drop missing glance images ([[phab:T228972|T228972]])
* 09:32 arturo: (codfw1dev) deleting a bunch of VMs that were running in now missing hypervisors
* 09:31 arturo: (codfw1dev) deleting a bunch of VMs in ERROR and SHUTDOWN state
* 09:27 arturo: last log entry refers to the codfw1dev deployment
* 09:27 arturo: cleanup `nova service-list` from old hypervisors (labtest*)
* 09:23 arturo: refreshed nova DB grants in clouddb2001-dev for the codfw1dev deployment
* 08:47 arturo: cleanup the cloud-announce pending emails (spam)

=== 2019-07-23 ===
* 19:43 andrewbogott: restarting rabbitmq-server on cloudcontrol1003 and 1004

=== 2019-07-22 ===
* 23:44 bd808: Restarted maintain-kubeusers on tools-k8s-master-01 ([[phab:T228529|T228529]])

=== 2019-07-11 ===
* 22:07 bd808: Ran `sudo systemctl stop designate_floating_ip_ptr_records_updater.service` on cloudcontrol1003
* 22:01 bd808: `sudo apt-get install python2.7-dbg` on cloudcontrol1003 to debug hung python process
* 21:48 bd808: Ran `sudo systemctl stop designate_floating_ip_ptr_records_updater.service` on cloudcontrol1004

=== 2019-06-25 ===
* 16:05 bstorm_: updated python3.4 to update4 wherever it was installed on Jessie VMs to prevent issues with broken update3.
* 14:56 bstorm_: Updated python 3.4 on the labs-puppetmaster server

=== 2019-06-03 ===
* 15:55 arturo: [[phab:T221769|T221769]] rebooting cloudservices1003 after bootstrapping is apparently completed

=== 2019-05-28 ===
* 21:42 bstorm_: unmounting labstore1003-scratch on all cloud clients
* 18:14 bstorm_: [[phab:T209527|T209527]] switched mounts from labstore1003 to cloudstore1008 for scratch

=== 2019-05-20 ===
* 17:25 arturo: [[phab:T223923|T223923]] dropped compat-network config from /etc/network/interfaces in eqiad1/codfw1dev neutron nodes
* 17:22 arturo: [[phab:T223923|T223923]] dropped br-compat bridges and vlan interfaces (1102 and 2102) in eqiad1/codfw1dev neutron nodes
* 17:07 arturo: [[phab:T223923|T223923]] dropped compat-network configuration from the neutron database in eqiad1
* 16:55 arturo: [[phab:T223923|T223923]] dropped compat-network configuration from the neutron database in codfw1dev

=== 2019-05-15 ===
* 17:00 andrewbogott: touching /root/firstboot_done on all VMs that cumin can reach.  This will prevent firstboot.sh from running a second time if/when any of these are rebooted.  [[phab:T223370|T223370]]

=== 2019-04-26 ===
* 15:51 arturo: andrew updated dns servers for the cloud-instances2-b-eqiad subnet in neutron: 208.80.154.143 and 208.80.154.24

=== 2019-04-25 ===
* 11:14 arturo: [[phab:T221760|T221760]] increased size of conntrack table

=== 2019-04-24 ===
* 12:54 arturo: [[phab:T220051|T220051]] puppet broken in every VM in Cloud VPS, fixing right now

=== 2019-04-22 ===
* 11:14 arturo: create by hand /var/cache/labsaliaser/labs-ip-aliases.json in cloudservices2002-dev ([[phab:T218575|T218575]])

=== 2019-04-16 ===
* 22:55 bd808: cloudcontrol2003-dev: added `exit 0` to /etc/cron.hourly/keystone to stop cron spam on partially configured cluster
* 12:08 arturo: rebooting cloudvirt200[123]-dev because deep changes in config
* 11:27 arturo: [[phab:T219626|T219626]] add DB grants for neutron and glnace to clouddb2001-dev (codfw1dev)
* 10:37 arturo: [[phab:T219626|T219626]] replace 208.80.153.75 with 208.80.153.59 in the clouddb2001-dev database (codfw1dev deployment)
* 10:30 arturo: [[phab:T219626|T219626]] replace labtestcontrol2003 with cloudcontrol2001-dev in the clouddb2001-dev database (codfw1dev deployment)

=== 2019-04-15 ===
* 13:08 arturo: [[phab:T219626|T219626]] add DB grants for keystone/nova/nova_api to clouddb2001-dev (codfw1dev)

=== 2019-04-13 ===
* 18:25 bd808: Restarted nova-compute service on cloudvirt1015 ([[phab:T220853|T220853]])

=== 2019-04-11 ===
* 12:00 arturo: [[phab:T151704|T151704]] deploying oidentd to cloudnet1xxx servers

=== 2019-04-02 ===
* 19:52 andrewbogott: installed new base Stretch image.  Updated packages, and runs apt-get dist-upgrade on first boot.

=== 2019-03-29 ===
* 14:34 andrewbogott: moving tools-static.wmflabs.org to point to tools-static-13 in eqiad1-r
* 00:00 bstorm_: [[phab:T193264|T193264]] Added osm.db.svc.eqiad.wmflabs to cloud DNS

=== 2019-03-25 ===
* 00:40 bd808: Restarted maintain-dbusers on labstore1004. Process hung up on failed LDAP connection.

=== 2019-03-21 ===
* 19:32 andrewbogott: restarting keystone on cloudcontrol1003

=== 2019-03-15 ===
* 16:00 gtirloni: increased nscd cache size ([[phab:T217280|T217280]])

=== 2019-03-14 ===
* 19:04 gtirloni: bstorm started nfsd on labstore1006 ([[phab:T218341|T218341]])
* 16:42 gtirloni: published new debian-9.8 image ([[phab:T218314|T218314]])

=== 2019-03-04 ===
* 19:37 bstorm_: umounted /mnt/nfs/dumps-labstore1006.wikimedia.org across all VPS projects for [[phab:T217473|T217473]]

=== 2019-02-26 ===
* 12:46 gtirloni: shutdown toolsbeta-sgegrid-master (cronspam)

=== 2019-02-25 ===
* 10:32 gtirloni: restarted nfsd on labstore1004

=== 2019-02-21 ===
* 09:09 gtirloni: restarted uwsgi-labspuppetbackend.service on labpuppetmaster1001
* 07:42 gtirloni: created project cloudstore
* 07:36 gtirloni: deleted wmcs-nfs project

=== 2019-02-20 ===
* 21:58 andrewbogott: silencing shinken and disabling puppet on shinken-02 for now

=== 2019-02-19 ===
* 12:00 gtirloni: added nagios@icinga2001.wikimedia.org to cloud-admin-feed@ allowed senders

=== 2019-02-18 ===
* 20:21 gtirloni: downtimed cloudvirt1020
* 20:12 gtirloni: ran `labs-ip-alias-dump.py` on cloudservices/labservices servers

=== 2019-02-15 ===
* 13:10 arturo: [[phab:T216239|T216239]] labvirt1019 has been drained
* 12:22 arturo: [[phab:T216239|T216239]] draining labvirt1009 with a command like this: `root@cloudcontrol1004:~# wmcs-cold-migrate --region eqiad --nova-db nova 2c0cf363-c7c3-42ad-94bd-{{Gerrit|e586f2492321}} labvirt1001`
* 12:02 arturo: more nova service cleanups in the database (labvirts that were reallocated to eqiad1)
* 11:34 arturo: [[phab:T216190|T216190]] cleanup from nova database `nova service-delete 35`
* 03:50 andrewbogott: updated VPS base images for Jessie and Stretch, now featuring Stretch 9.7

=== 2019-02-11 ===
* 18:13 gtirloni: cleaned old metrics data in labmon1001 [[phab:T215417|T215417]]
* 15:28 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1011
* 14:18 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1010

=== 2019-02-08 ===
* 14:56 gtirloni: running `maintain-views --all-databases --replace-all` on labsdb1009

=== 2019-02-06 ===
* 11:47 gtirloni: downtimed labmon100{1,2} [[phab:T215399|T215399]]
* 00:17 bstorm_: [[phab:T214106|T214106]] deleted bstorm-test2 project to clean up

=== 2019-02-05 ===
* 10:48 arturo: labmon1001 is now part of the 'eqiad1-r' region

=== 2019-02-01 ===
* 09:54 arturo: moving canary1015-01 VM instance from cloudvirt1024 back to cloudvirt1015

=== 2019-01-31 ===
* 12:44 arturo: [[phab:T215012|T215012]] depooling cloudvirt1015 and migrating all VMs to cloudvirt1024

=== 2019-01-25 ===
* 20:11 gtirloni: deleted project yandex-proxy [[phab:T212306|T212306]]
* 20:11 gtirloni: deleted project [[phab:T212306|T212306]]

=== 2019-01-24 ===
* 11:50 arturo: [[phab:T213925|T213925]] modify subnet cloud-instances-transport1-b-eqiad1 to avoid floating IP allocations from here
* 11:07 arturo: [[phab:T214299|T214299]] failover cloudnet1003 to cloudnet1004
* 10:03 arturo: [[phab:T214299|T214299]] reimage cloudnet1004 to debian stretch
* 09:51 arturo: [[phab:T214299|T214299]] failover cloudnet1004 to cloudnet1003

=== 2019-01-22 ===
* 19:19 arturo: [[phab:T214299|T214299]] stretch cloudnet1003 is apparently all set
* 18:40 arturo: [[phab:T214299|T214299]] manually delete from neutron agents from cloudnet1003 (must be added again after reimage, with new uuids)
* 18:37 arturo: [[phab:T214299|T214299]] reimaging cloudnet1003 as debian stretch
* 17:35 jbond42: starting roll out of apt package updates to
* 14:41 gtirloni: [[phab:T214369|T214369]] deployed new jessie and stretch VM images

=== 2019-01-21 ===
* 18:29 gtirloni: installed libguestfs-tools on cloudvirt1021

=== 2019-01-16 ===
* 14:21 andrewbogott: stopping old VPS proxies in eqiad — [[phab:T213540|T213540]]

=== 2019-01-15 ===
* 14:20 andrewbogott: changing tools.wmflabs.org to point to tools-proxy-03 in eqiad1

=== 2019-01-13 ===
* 20:00 andrewbogott: VPS proxies are now running in eqiad1 on proxy-01.  Old VMs will wait a bit for deletion.  [[phab:T213540|T213540]]
* 19:12 andrewbogott: moving the VPS proxy API backend to proxy-01.project-proxy.eqiad.wmflabs, as per [[phab:T213540|T213540]]
* 17:11 andrewbogott: moving all VPS dynamic proxies to proxy-eqiad1.wmflabs.org aka proxy-01.project-proxy.eqiad.wmflabs, as per [[phab:T213540|T213540]]

=== 2019-01-09 ===
* 22:21 bd808: neutron quota-update --tenant-id tools --port 256

=== 2019-01-08 ===
* 18:59 bd808: Definately did NOT delete uid=novaadmin,ou=people,dc=wikimedia,dc=org
* 18:59 bd808: Deleted LDAP user uid=neutron,ou=people,dc=wikimedia,dc=org
* 18:58 bd808: Deleted LDAP user uid=novaadmin,ou=people,dc=wikimedia,dc=org

=== 2019-01-06 ===
* 22:03 bd808: Set floatingip quota of 60 for tools project in eqiad1-r region ([[phab:T212360|T212360]])

=== 2018-12-20 ===
* 17:10 arturo: [[phab:T207663|T207663]] renumbered transport network in eqiad1

=== 2018-12-05 ===
* 17:59 arturo: [[phab:T207663|T207663]] changed labtestn transport network addressing from private to public

=== 2018-12-03 ===
* 13:25 arturo: [[phab:T202886|T202886]] create again PTR records after dnsleak.py fix

=== 2018-11-30 ===
* 14:08 arturo: running dns leaks cleanup `root@cloudcontrol1003:~# /root/novastats/dnsleaks.py --delete`

=== 2018-11-28 ===
* 17:33 gtirloni: deleted contintcloud project ([[phab:T209644|T209644]])

=== 2018-11-27 ===
* 13:32 gtirloni: enabled DRBD stats collection on labstore100[4-5] [[phab:T208446|T208446]]

=== 2018-11-22 ===
* 07:12 gtirloni: deployed new debian-9.6-stretch image

=== 2018-11-21 ===
* 10:48 arturo: re-created compat-net as not shared in labtestn to test stuff related to [[phab:T209954|T209954]]

=== 2018-11-16 ===
* 12:43 gtirloni: armed keyholder on labpuppetmaster1001/1002 after reboots
* 12:08 gtirloni: rebooted labpuppetmaster1001 ([[phab:T207377|T207377]])
* 11:57 gtirloni: rebooted labpuppetmaster1002 ([[phab:T207377|T207377]])

=== 2018-11-14 ===
* 17:19 gtirloni: added cloudvirt1016 to scheduler pool ([[phab:T209426|T209426]])
* 15:41 gtirloni: reimaging labvirt1016 as cloudvirt1016
* 15:14 gtirloni: reset-failed systemd unit nova-scheduler on cloudcontrol1004
* 13:52 gtirloni: rebooted labservices1002 after package upgrades ([[phab:T207377|T207377]])
* 13:23 gtirloni: rebooted labstore2004 after package upgrades ([[phab:T207377|T207377]])
* 13:20 gtirloni: rebooted labstore2003 after package upgrades ([[phab:T207377|T207377]])
* 13:20 gtirloni: rebooted labstore2001/labstore2003 after package upgrades ([[phab:T207377|T207377]])
* 12:08 gtirloni: rebooted labnet1002 after package upgrades
* 12:01 gtirloni: rebooted labmon1002 after package upgrades
* 11:41 gtirloni: rebooted labcontrol1002 after package upgrades
* 11:15 gtirloni: rebooted cloudcontrol1004 after package upgrades

=== 2018-11-09 ===
* 18:17 gtirloni: restarted neutron-linuxbridge-agent on cloudvirt1018/1023

=== 2018-11-08 ===
* 11:00 gtirloni: Added novaproxy-02 to $CACHES
* 10:50 gtirloni: Added cloudvirt1017 to eqiad1 region

=== 2018-11-07 ===
* 13:49 arturo: [[phab:T208733|T208733]] moving labvirt1017 from main deployment to eqiad1 and renaming it to cloudvirt1017

=== 2018-10-22 ===
* 16:24 arturo: [[phab:T206261|T206261]] another update to dmz_cidr in eqiad1
* 10:26 arturo: change again in dmz_cidr in eqiad1: VMs will connect between them without NAT even when using floating IPs ([[phab:T206261|T206261]])

=== 2018-10-19 ===
* 12:02 arturo: revert change in dmz_cidr in eqiad1 for now ([[phab:T206261|T206261]])
* 11:16 arturo: change in dmz_cidr in eqiad1: VMs will connect between them without NAT even when using floating IPs ([[phab:T206261|T206261]])
* 10:14 arturo: we have new virt servers in the eqiad1 deployment since past week and this week: cloudvirt1018, cloudvirt1023, cloudvirt1024

=== 2018-09-26 ===
* 10:40 arturo: [[phab:T205524|T205524]] all sorts of restarts in all neutron daemons
* 10:20 arturo: [[phab:T205524|T205524]] stop/start all neutron agents in cloudnet1003.eqiad.wmnet
* 10:13 arturo: [[phab:T205524|T205524]] restart all agents in cloudnet1004.eqiad.wmnet
* 10:10 arturo: restart neutron-server in cloudcontrol1003, investigating [[phab:T205524|T205524]]

=== 2018-09-24 ===
* 10:57 arturo: try to increase floating ip allocation pool in eqiad1. Of 185.15.56.0/25 we are using only 185.15.56.10-185.15.56.31, I don't know why. Let's use 185.15.56.2-185.15.56.126

=== 2018-09-21 ===
* 17:18 bd808: Running `sudo maintain-meta_p --all-databases --purge` across labsdb10(09{{!}}10{{!}}11) for [[phab:T201890|T201890]]

=== 2018-09-17 ===
* 22:08 bd808: Granted gtirloni project roles of admin, projectadmin, and user

=== 2018-09-12 ===
* 11:20 arturo: [[phab:T202636|T202636]] distributing default routes using classless-static-route for all VMs in main/labtest (dnsmasq/nova-network)

=== 2018-09-11 ===
* 16:52 arturo: again, restarted nova-network after killing all dnsmasq procs in labnet1001 for [[phab:T202636|T202636]]
* 16:08 arturo: restarted nova-network after killing all dnsmasq procs in labnet1001 for [[phab:T202636|T202636]]
* 10:53 arturo: [[phab:T202636|T202636]] creating all the compat-network configuration in neutron
* 10:36 arturo: [[phab:T202636|T202636]] creating br-compat bridge in eqiad1 for the compat network
* 10:33 arturo: [[phab:T202636|T202636]] manually reserve 10.68.23.253 (in nova-network)

=== 2018-09-10 ===
* 22:46 andrewbogott: deleting all VMs on labvirt1019 and 1020 as prep for [[phab:T204003|T204003]]

=== 2018-08-30 ===
* 15:46 andrewbogott: restarting rabbitmq-server on cloudcontrol1003
* 13:07 arturo: [[phab:T202636|T202636]] internal network routing now exists in labtest/labtestn for VM to communicate with each other

=== 2018-08-28 ===
* 11:04 arturo: [[phab:T202549|T202549]] eqiad1 databases are all now running in m5-master. Mysql has been cleaned from cloudcontrol100[3,4]

=== 2018-08-23 ===
* 16:17 arturo: [[phab:T188589|T188589]] bstorm_ merged patch to reduce nova DB connection usage
* 13:15 arturo: [[phab:T202115|T202115]] `root@cloudcontrol1003:~# neutron subnet-update --allocation-pool start=10.64.22.4,end=10.64.22.4 e4fb2771-a361-4add-ac4e-280cc300c59f`
* 13:10 arturo: [[phab:T202115|T202115]] (was `{&quot;start&quot;: &quot;10.64.22.2&quot;, &quot;end&quot;: &quot;10.64.22.254&quot;}` )
* 13:08 arturo: [[phab:T202115|T202115]] `root@cloudcontrol1003:~# neutron subnet-update --allocation-pool start=10.64.22.254,end=10.64.22.254 e4fb2771-a361-4add-ac4e-280cc300c59f`

=== 2018-08-22 ===
* 15:28 arturo: cleanup local glance,keystone databases in cloudcontrol1003.wikimedia.org (already in m5-master)
* 15:27 arturo: cleanup local keystone database in cloudcontrol1003.wikimedia.org (already in m5-master)

=== 2018-08-21 ===
* 15:39 andrewbogott: initial test message
* 10:31 arturo: eqiad1 remove leftover port for HA on labnet1004
* 10:15 arturo: test

=== 2018-05-07 ===
* 18:07 bstorm_: stopped the toolhistory job because it is totally broken and fills /tmp.

=== 2018-02-09 ===
* 00:55 bd808: Added Arturo Borrero Gonzalez and Bstorm as project members
* 00:54 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]])

{{SAL|Project Name=admin}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>ckvspp2hhoqjvh322rldyvuvoqj7zbw</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.noclaims/SAL</title>
    <ns>498</ns>
    <id>313327</id>
    <revision>
      <id>2309620</id>
      <parentid>2174750</parentid>
      <timestamp>2025-06-08T10:17:32Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~multichill@tools-bastion-12: Switched to BotPasswords</comment>
      <origin>2309620</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="960" sha1="9apjip8awh3btlfa4wpq9dga3u5shph" xml:space="preserve">=== 2025-06-08 ===
* 10:17 wmbot~multichill@tools-bastion-12: Switched to BotPasswords

=== 2024-05-03 ===
* 09:33 wmbot~multichill@tools-bastion-12: Deployed jobs.yml with claim-templates hourly

=== 2024-03-17 ===
* 18:41 wmbot~multichill@tools-sgebastion-11: Recovering the bot

=== 2020-02-26 ===
* 15:47 wm-bot: &lt;root&gt; Migrated to 2020 Kubernetes cluster

=== 2016-12-17 ===
* 22:03 multichill: Added .lighttpd.conf and webservice restart so that logs are now send as &quot;text/plain;charset=UTF-8&quot;
* 20:55 multichill: Moved the two jobs here (one in the morning and one in the evening) and updated https://www.wikidata.org/wiki/User:NoclaimsBot
* 20:11 multichill: Set up the bot with a clone of https://github.com/multichill/toollabs and a symlinked pywikibot (git clone is broken see phab:[[phab:T151351|T151351]] )

=== 2016-12-15 ===
* 20:09 yuvipanda: move webservice to kubernetes, is only serving static files


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>9apjip8awh3btlfa4wpq9dga3u5shph</sha1>
    </revision>
    <revision>
      <id>2309622</id>
      <parentid>2309620</parentid>
      <timestamp>2025-06-08T10:55:19Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~multichill@tools-bastion-12: Switched to BotPasswords for T395205</comment>
      <origin>2309622</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1057" sha1="00uy68410gbmdznz66farahddgwm0xy" xml:space="preserve">=== 2025-06-08 ===
* 10:55 wmbot~multichill@tools-bastion-12: Switched to BotPasswords for [[phab:T395205|T395205]]
* 10:17 wmbot~multichill@tools-bastion-12: Switched to BotPasswords

=== 2024-05-03 ===
* 09:33 wmbot~multichill@tools-bastion-12: Deployed jobs.yml with claim-templates hourly

=== 2024-03-17 ===
* 18:41 wmbot~multichill@tools-sgebastion-11: Recovering the bot

=== 2020-02-26 ===
* 15:47 wm-bot: &lt;root&gt; Migrated to 2020 Kubernetes cluster

=== 2016-12-17 ===
* 22:03 multichill: Added .lighttpd.conf and webservice restart so that logs are now send as &quot;text/plain;charset=UTF-8&quot;
* 20:55 multichill: Moved the two jobs here (one in the morning and one in the evening) and updated https://www.wikidata.org/wiki/User:NoclaimsBot
* 20:11 multichill: Set up the bot with a clone of https://github.com/multichill/toollabs and a symlinked pywikibot (git clone is broken see phab:[[phab:T151351|T151351]] )

=== 2016-12-15 ===
* 20:09 yuvipanda: move webservice to kubernetes, is only serving static files


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>00uy68410gbmdznz66farahddgwm0xy</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.multichill/SAL</title>
    <ns>498</ns>
    <id>443599</id>
    <revision>
      <id>2309623</id>
      <parentid>2298025</parentid>
      <timestamp>2025-06-08T11:12:35Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~multichill@tools-bastion-12: Switched BotMultichillT to BotPasswords for T395205</comment>
      <origin>2309623</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1277" sha1="npw5lxsb2c7h9l6quz2rvsx1jxj2c7a" xml:space="preserve">=== 2025-06-08 ===
* 11:12 wmbot~multichill@tools-bastion-12: Switched BotMultichillT to BotPasswords for [[phab:T395205|T395205]]

=== 2025-05-04 ===
* 15:59 wmbot~multichill@tools-bastion-12: public-domain-paintings added
* 14:31 wmbot~multichill@tools-bastion-12: and also added christian-paintings
* 14:31 wmbot~multichill@tools-bastion-12: Added religious-paintings

=== 2024-11-26 ===
* 18:27 wmbot~multichill@tools-bastion-12: Tired of the Unable to start, out of quota for memory, memory, created [[phab:T380902|T380902]] for more memory

=== 2024-05-21 ===
* 17:01 wmbot~multichill@tools-bastion-12: deployed infobox-missing-commons-category to empty out https://commons.wikimedia.org/wiki/Category:Uses_of_Wikidata_Infobox_missing_Commons_Category_statement

=== 2024-05-03 ===
* 17:37 wmbot~multichill@tools-bastion-12: Deployed jobs.yml with quality-image-add daily [[phab:T319912|T319912]]

=== 2024-03-17 ===
* 17:11 wmbot~multichill@tools-sgebastion-11: webservice started again

=== 2022-06-04 ===
* 17:22 wm-bot: &lt;multichill&gt; New virtualenv. Pull on pywikibot and other code. First jobs running again on Buster

=== 2019-03-04 ===
* 20:37 andrewbogott: killed find_paint because it was filling up /tmp on its exec node


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>npw5lxsb2c7h9l6quz2rvsx1jxj2c7a</sha1>
    </revision>
  </page>
  <page>
    <title>Map of database maintenance</title>
    <ns>0</ns>
    <id>449160</id>
    <revision>
      <id>2309618</id>
      <parentid>2309598</parentid>
      <timestamp>2025-06-08T00:02:06Z</timestamp>
      <contributor>
        <username>Dexbot</username>
        <id>30554</id>
      </contributor>
      <minor />
      <comment>Bot: Updating the report</comment>
      <origin>2309618</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="3493" sha1="2ol5tvtqw8qpox6qept7tkxw9yo57cq" xml:space="preserve">{{/Header}}
== Today (2025-06-08) ==
== Yesterday (2025-06-07) ==
== Last seven days ==
{| class=&quot;wikitable&quot;
|+ eqiad
|-
! Section !! Work
|-
| es6 || [[phab:T395867|Switchover es6 master (es1037 -&amp;gt; es1038) (T395867)]] (marostegui)
|-
| es7 || 
* [[phab:T395647|Migrate es7 to MariaDB 10.11 (T395647)]] (marostegui)
* [[phab:T395982|Switchover es7 master (es1035 -&amp;gt; es1039) (T395982)]] (marostegui)
|-
| pc1 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc2 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc3 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc4 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc5 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc6 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc7 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc8 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| s1 || [[phab:T395663|MariaDB 10.11.13 released (T395663)]] (marostegui)
|-
| s2 || [[phab:T395241|Login (T395241)]] (fceratto)
|-
| s6 || [[phab:T383795|Move sX to STATEMENT based replication (T383795)]] (marostegui)
|-
| s8 || [[phab:T395241|Login (T395241)]] (fceratto)
|-
|}
{| class=&quot;wikitable&quot;
|+ codfw
|-
! Section !! Work
|-
| es6 || 
* [[phab:T395420|Switchover es6 master (es2037 -&amp;gt; es2035) (T395420)]] (marostegui)
* [[phab:T395771|Productionize es2047, es2048, es1047, es1048 (T395771)]] (marostegui)
|-
| es7 || 
* [[phab:T395647|Migrate es7 to MariaDB 10.11 (T395647)]] (marostegui)
* [[phab:T395771|Productionize es2047, es2048, es1047, es1048 (T395771)]] (marostegui)
* [[phab:T395785|Switchover es7 master (es2038 -&amp;gt; es2039) (T395785)]] (marostegui)
|-
| pc1 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc2 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc3 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc4 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc5 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc6 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc7 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| pc8 || [[phab:T395983|Migrate /srv/sqldata-cache directory in parsercache to /srv/sqldata (T395983)]] (marostegui)
|-
| s2 || [[phab:T383795|Move sX to STATEMENT based replication (T383795)]] (marostegui)
|-
| s4 || [[phab:T395241|Login (T395241)]] (fceratto)
|-
| s6 || 
* [[phab:T383795|Move sX to STATEMENT based replication (T383795)]] (marostegui)
* [[phab:T395989|Migrate s6 to MariaDB 10.11 (T395989)]] (marostegui)
|-
| s8 || [[phab:T395241|Login (T395241)]] (fceratto)
|-
|}


[[Category:MariaDB]]</text>
      <sha1>2ol5tvtqw8qpox6qept7tkxw9yo57cq</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.geograph/SAL</title>
    <ns>498</ns>
    <id>450811</id>
    <revision>
      <id>2309621</id>
      <parentid>2281291</parentid>
      <timestamp>2025-06-08T10:53:52Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~multichill@tools-bastion-12: Switch to botpasswords for T395205</comment>
      <origin>2309621</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1159" sha1="rq1757jayca8o2qzlky2uw9xhwyczdk" xml:space="preserve">=== 2025-06-08 ===
* 10:53 wmbot~multichill@tools-bastion-12: Switch to botpasswords for [[phab:T395205|T395205]]

=== 2025-03-10 ===
* 18:41 wmbot~multichill@tools-bastion-12: Unable to do tools.geograph@tools-bastion-12:~$ toolforge jobs restart geograph-uploader-resumed, keeps giving requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=30)
* 17:28 wmbot~multichill@tools-bastion-12: Unable to do tools.geograph@tools-bastion-12:~$ toolforge jobs restart geograph-uploader-resumed, keeps giving requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=30)

=== 2024-05-03 ===
* 11:56 wmbot~multichill@tools-bastion-12: [[phab:T319765|T319765]] Deployed jobs.yml with geograph-uploader-resumed hourly

=== 2023-09-14 ===
* 17:48 taavi: tools-sgegrid-master:~ $ sudo qdel -f {{Gerrit|2141868}}

=== 2022-06-04 ===
* 17:18 wm-bot: &lt;multichill&gt; New virtualenv for Buster. Stuck with stupid venv because of missing mwparserfromhell/wikitextparser


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>rq1757jayca8o2qzlky2uw9xhwyczdk</sha1>
    </revision>
  </page>
  <page>
    <title>Talk:Robot policy</title>
    <ns>1</ns>
    <id>458664</id>
    <revision>
      <id>2309613</id>
      <parentid>2293619</parentid>
      <timestamp>2025-06-07T18:24:04Z</timestamp>
      <contributor>
        <username>Dušan Kreheľ</username>
        <id>26659</id>
      </contributor>
      <comment>/* Year update 2025 */ new section</comment>
      <origin>2309613</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="7405" sha1="4pquykq4ty3zhnr6xq72qfrcisv1sgz" xml:space="preserve">== The intro is a bit harsh. Bots are our most productive user class. Please make it friendlier! ==

There's no need to lead off with &quot;Failing to comply with these rules will generally result in... you as an Operator eventually banned from accessing our resources&quot; or the all caps &quot;'''Failure to follow these guidelines may result in your bot being blocked or heavily rate-limited.'''&quot;

- first try prolific light rate-limits before scaring people about blocking and heavy limits :)
- clarify that rate-limits come first, and how bots will be notified that they are being limited 


Currently the page mentions specifics that may conflict with the operation of useful good-faith low-volume bots. Clarify that this is a dialogue to keep the sites running smoothly, feedback welcome to tune the requirements, &amp;c.  e.g.:

 &quot;Bots that don't follow these guidelines may be rate-limited
  ... and informed of the reason, and pointed to ways to request higher quotas or exemptions
  Bots that repeatedly try to get around these guidelines or rate limits may be blocked&quot;

[[User:Sj|Sj]] ([[User talk:Sj|talk]]) 16:26, 9 April 2025 (UTC)

:@[[User:Sj|Sj]]: Bots are, by far, our most expensive users overall. A tiny sliver of them are run by editors who are here to contribute edits or support editors, which are the uses you're thinking of when you say &quot;bots&quot;, but that's not what's meant here. [[User:Jdforrester (WMF)|Jdforrester (WMF)]] ([[User talk:Jdforrester (WMF)|talk]]) 17:32, 9 April 2025 (UTC)

:: I don't think that the tone of language on this page is going to affect the most expensive subsets of robots affected by this policy.  It will affect the good-faith devs who know enough to find this page in the first place.
:: I should very much like to know the classes of bots and their distribution, including abusive bots [which we never needed a detailed policy like this to block], misconfigured ones, trained ones, those belonging to the emperor, &amp;c.  Many of them are devoted to disseminating knowledge, and understanding how and why is among other things a way of identifying new knowledge flows by following demand. In this context, given related threads over the past year, I think first of bots and agents run by popular apps and AI services with millions of users, whose organizations tend to be supporters of the wikiverse, while also operating at high volumes and not yet being in equilibrium with their ecosystem.
:: The fact that this page jumped from no mention of rate-limits at all to leading off with [permanent] banning and blocking suggests a spikiness of internal policy that doesn't need to leak through into public communication.  There's always time to convey context, and make it a warm dialogue.  [[User:Sj|Sj]] ([[User talk:Sj|talk]]) 22:10, 9 April 2025 (UTC)
:::I agree with you that the tone of that part is a bit too harsh, it's clear from some of the feedback I got. I think your proposed wording works better.
:::But please rest assured, abusive bots are already being pointed to this policy and this policy is mostly intended for them. [[User:GLavagetto (WMF)|GLavagetto (WMF)]] ([[User talk:GLavagetto (WMF)|talk]]) 04:51, 11 April 2025 (UTC)
::::Also on why we need this change in the policy: the SRE team has been playing whack-a-mole for the best part of the last two years; we've even purpose-built systems to make filtering of traffic easier and faster. 
::::And it's getting worse: just in the last month, we've had to add 9 filtering rules for new abusive bots. Which means not only 9 incident responses, but also a lot of work to make sure we eventually remove filters that are not needed and might cause some false positives.
::::The only way out of this - which again i don't think is sustainable - is to move from episodic enforcements of our rules to systematic enforcement.
::::That's also why I tried to cover more systems in the updated policy, including media files and e.g. phabricator.
::::[[User:GLavagetto (WMF)|GLavagetto (WMF)]] ([[User talk:GLavagetto (WMF)|talk]]) 06:01, 11 April 2025 (UTC)

== Consolidating overlapping pages ==

@[[User:GLavagetto (WMF)|GLavagetto (WMF)]] thank you for these major documentation improvements! 

I just noticed the page [[bot traffic]], which is linked from https://developer.wikimedia.org/use-content/bot-traffic/. Since it only has instructions for indicating an operator's IP ranges, it seems unnecessarily complicated for it to be on a separate page. Why don't we merge that page into this one?  [[User:Neil Shah-Quinn (WMF)|Neil Shah-Quinn (WMF)]] ([[User talk:Neil Shah-Quinn (WMF)|talk]]) 20:41, 11 April 2025 (UTC)

:The reason that page is separated is that, eventually, we want to have a proper intake process with a form, where the content of that page should go to. Having it on wikitech right now is just a placeholder while we figure out a proper process for handling requests at scale. [[User:GLavagetto (WMF)|GLavagetto (WMF)]] ([[User talk:GLavagetto (WMF)|talk]]) 13:49, 14 April 2025 (UTC)

Actually, along the same lines, what do think about moving the content on this page onto the [https://developer.wikimedia.org/use-content/bot-traffic/ developer portal page]? I feel like this would have a number of benefits:
* The page would look more official and serious, whereas a normal wiki page like this looks more like documentation
* Readers would see the concrete advice on this page first, whereas the developer portal currently points first to the more formal, wordy pages on Governance Wiki
* Avoids frictionless editing of the page (normally, that's a feature, not a bug, but I think a serious, high-visibility policy like this is an exception).

It does look like it might be difficult to write a detailed page within the architecture of the developer portal, so if that's the case, maybe the best compromise would be to make the developer portal page link ''only'' to this page. This page already links to the other pages as relevant, so few changes would be needed. We could then protect this page to get the same benefit of a speed bump for edits (it doesn't need to be super strict; I think limiting it to auto-confirmed users would be plenty). [[User:Neil Shah-Quinn (WMF)|Neil Shah-Quinn (WMF)]] ([[User talk:Neil Shah-Quinn (WMF)|talk]]) 20:57, 11 April 2025 (UTC)

:The developer portal is deliberately designed to provide [[mw:Developer_Portal#User_journeys|user journey]] focused links to on-wiki documentation. The particular content you are wondering about changing was added in [[phab:T388051]] based on the needs of the https://enterprise.wikimedia.com/ team. -- [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]]) 00:27, 16 April 2025 (UTC)

== Year update 2025 ==

'''Proti.''' Against. I miss the following scenario in the rules (it comes from practice with my bot): I have a bot, I make changes in 1 thread, I want an action like download-[edit-save]-wait_X_s . It seems to me that the new rules are unnecessarily complicated for this scenario. (automatic translate, original: Chybí mi v pravidlách uvedenie nasledujúceho scenára (vychadzá z praxe s mojím botom): Mám bota, zmeny vykonávam v 1 vlákne, chcem činnosť v štýle stiahnem-[upravím-uložím]-počkám_X_s . Príde mi, že na tento scenár sú nové pravidla zbytočne zložité. [[User:Dušan Kreheľ|Dušan Kreheľ]] ([[User talk:Dušan Kreheľ|talk]]) 18:24, 7 June 2025 (UTC)</text>
      <sha1>4pquykq4ty3zhnr6xq72qfrcisv1sgz</sha1>
    </revision>
  </page>
</mediawiki>