Software Update Runbooks
5 items
Key Rotation Runbooks
5 items
Software Updates Overview
Overview of all AIsware software update procedures.
Key Rotation Overview
This document is the entry point for all AIsware key rotation procedures. It describes which key types exist, their rotation schedule, and which rotations require downtime. The runbooks for those procedures are linked below.
VPN Agent Reconciliation Failure
Responding to the VPN Agent Reconciliation Failure Grafana alert.
Apply IP Whitelist Changes
This script handles all the necessary steps to ship a new iplist file to the OpenWRT firewall configuration on the router (172.20.0.1). It diffs the incoming file against the one currently deployed on the router, copies it over on confirmation, and triggers a live reload of the ipsets.
CAPI cluster health alerts
Responding to CAPI cluster health and provisioning alerts
Ceph node maintenance
Safe procedure for taking a Ceph storage node offline and returning it to service
Deleting orphaned tenant clusters
Deleting orphaned tenant clusters
Getting provisioning logs
Getting provisioning logs
Hedgehog switch credentials update
Updating switch user credentials in the Hedgehog fabricator spec
Hedgehog VM credentials update
Updating the core user password and SSH authorized keys on the Hedgehog control node VM
Management Cluster Subnet Migration
Overview
Migrating LVM OSDs to Raw BlueStore
Migrating LVM-backed OSDs to raw BlueStore specs
Neutron L3 agent recovery
Responding to the Neutron Service Availability Degraded alert caused by a dead or stuck neutron-l3-agent
OpenStack cluster safe stop / cold start
Shut down the Phoenix OpenStack cluster cleanly and bring it back up without data loss.
Personal data disposal procedure
Removing personal data for GDPR erasure request
pgpool stuck on wrong primary after PostgreSQL HA failover
Recovering from pgpool refusing database connections after a PostgreSQL HA failover is cancelled mid-flight.
Recovering a locked RBD volume
Unlock a Ceph RBD volume stuck after a VM hard crash
Router Host Log Access
Getting logs from the router
Switch heartbeat alert
Responding to Switch Heartbeat Evaluation alerts caused by a hedgehog-agent crash loop
Tenant service termination — data disposal procedure
Removing all tenant data upon service termination.