Skip to main content
Version: v25.1.5

Operator Overview

Operator Overview for AIsware

This is the reference sheet for AIsware, an end-to-end solution to operate private, multi-tenant AI factories. Operators will find below an overview of the materials, infrastructure, and other requirements, and an entry point to the procedure to provision and configure the system.

Installer links, deployment files, and any other needed assets will be provided to clients directly. You can request them using the support@midokura.com email address.

System requirements​

note

Documentation files referenced here are provided in a downloadable artefact included in the hardware setup section.

  • Before proceeding, operators are expected to ensure that the underlying infrastructure meets the system requirements listed below.
  • Operating system requirements for the OpenStack control nodes are available in OS_REQUIREMENTS
  • Operators are expected to set up their hardware according to our official Blueprint, specifically with regard to network configuration, port and interface assignment.
    • Base Operating System for OSt controllers should be ubuntu-24.04
  • Storage. Operators are expected to provide a Ceph cluster, integrated in the infrastructure as defined in the blueprint. See more details in the Hardware Setup
  • Set up OAuth application(s) for SSO. Supported providers are Google (GOOGLE_SSO_SETUP) and Azure (AZURE_SSO_SETUP). See the Software Installation section for details.
  • Set up credentials for the private registry at ghcr.io/midokura. We will provide you with this token via secure means, and it will be required during the software installation process. More info at GHCR_AUTHENTICATION.

Overview​

The sections below cover the full provisioning process, split into hardware setup (racking, OS installation, network fabric, storage) and software installation (deploying the control plane via Ansible playbooks from the bastion node).

Hardware Setup​

The hardware setup covers all physical and foundational infrastructure steps required before deploying the control plane. Build the inventory file (inventory.yml) progressively as you complete each step, using the included inventory.example.yml as your starting point.

  1. Rack and cable hardware following the official Blueprint — pay particular attention to network topology, port and interface assignment, and storage cabling.

  2. Configure server BIOS and BMC — apply BMC account and SNMP settings, configure Secure Boot, and apply model-specific BIOS settings as described in BIOS_BMC_SETUP.

  3. Install OS on OpenStack control nodes — Ubuntu 24.04 with RAID1 disks, VLAN interfaces, IOMMU, and required packages as specified in OS_REQUIREMENTS.

  4. Set up the Router Box — configure BIOS (AMT, Secure Boot), flash Ubuntu 24.04 to the machine, and run the unattended cloud-init install as described in ROUTER_BOX_SETUP.

  5. Create the bastion0 VM on the router box — the KVM virtual machine that serves as the deployment host for all subsequent steps. See the ROUTER_BOX_SETUP bastion VM section.

  6. Prepare the bastion for deployment — before running the bootstrap script, the bastion must have the release package, a container runtime, and valid credentials in place. SSH into bastion0 and complete the following:

    • Download and extract the release package: 
      curl <artifact-url> -O release-assets.tar.gz
      mkdir release-assets && tar -xzf release-assets.tar.gz -C release-assets
      cd release-assets
      sha256sum -c SHA256SUMS
    • Install Podman (container runtime used by all deployment scripts): 
      sudo apt-get update && sudo apt-get install -y podman
    • Authenticate to GHCR — the deployment scripts pull the Ansible container from ghcr.io/midokura. Log in using a GitHub Personal Access Token with read:packages scope (see GHCR_AUTHENTICATION): 
      echo $CR_PAT | podman login ghcr.io -u USERNAME --password-stdin
    • Copy inventory.yml into the release-assets/ directory on the bastion.
    • Ensure SSH keys are present in ~/.ssh/ on the bastion — Ansible uses these to access cluster nodes (ansible_ssh_private_key_file in inventory.yml).
    • Have the vault password ready — the bootstrap script prompts for it at startup.

    See DEPLOYMENT for full details on each prerequisite.

  7. Bootstrap the network environment — provisions the OpenWRT router VM, PXE/TFTP server, local Docker registry, and HedgeHog controller VM. All commands run from the release-assets/ directory on bastion0. Run in two phases with a manual credential-change step in between.

    tip

    Run all bootstrap commands inside a tmux or screen session. The bootstrap takes a long time (HedgeHog controller installation alone can take ~100 minutes) and an SSH session drop will kill the process mid-run.

    tmux new -s bootstrap

    If you get disconnected, reattach with tmux attach -t bootstrap.

    caution

    If the main router is not yet configured or is being reconfigured, OpenWRT will take over as the network router during this step and deployment0 will lose internet access. If this happens, update the default gateway on deployment0 to the secondary router IP:

    sudo vim /etc/netplan/50-cloud-init.yaml  # change gateway IP from primary to secondary router
    sudo netplan apply
    note

    The Router Box requires python3-lxml to be installed (included in the cloud-init packages list in ROUTER_BOX_SETUP). If the router box was provisioned without it, install it manually before running the bootstrap:

    sudo apt-get install -y python3-lxml
    note

    Before running bootstrap, configure these required credential variables in your inventory. Bootstrap will fail with a clear error if any are missing:

    • hedgehog.login_password — new password for the core user on the VM (vault-encrypt in production)
    • hedgehog.authorized_keys — list of SSH keys to authorize on the VM; each entry is {name, key}
    • hedgehog.switch_users — admin and op switch user credentials (password + authorized keys for each)

    See Router Services Configuration for the full variable schema and examples.

    Phase 1 — deploy the HedgeHog controller VM (skips fabric provisioning):

    ./scripts/platform-setup.sh --bootstrap --skip-tags hedgehog-fabric

    Credential rotation runs automatically: operator SSH keys are installed using the ISO default password, the core user password is then changed and verified, and the Fabricator CRD is patched with operator-defined credentials — all without manual intervention.

    Phase 2 — provision the HedgeHog fabric:

    ./scripts/platform-setup.sh --bootstrap --tags hedgehog-fabric

    Switches boot into ONIE and are provisioned with the credentials already configured in the Fabricator CRD during Phase 1.

    note

    If Phase 1 times out during the Kubernetes API readiness check, re-run with both tags — the hedgehog_controller role is idempotent (VM already exists, so creation steps are skipped) and only the readiness checks and credential rotation re-run:

    ./scripts/platform-setup.sh --bootstrap --tags hedgehog-controller,hedgehog-fabric

    See ROUTER_BOX_CONFIGURATION for configuration details.

  8. Set up the network fabric — download the HedgeHog control node ISO, create the control VM, apply the fabric configuration, boot switches into ONIE, and install SONiC via HedgeHog auto-discovery. Follow all steps in NETWORK_CONTROL_NODE_SETUP.

  9. Provision the Ceph cluster — storage nodes must have Ubuntu installed and be reachable via SSH from the bastion before this step:

    • Run platform-setup.sh --provision-ceph to provision the cluster and generate keyrings, then promote them to ./keyrings/
    • Complete the RADOS Gateway setup: start the gateway service and create the admin user before deploying OpenStack; configure Keystone integration after OpenStack is deployed
    • See CEPH_SETUP for the full procedure.

Software Installation​

The software installation covers all steps to deploy and configure the control plane. Complete the hardware setup above before proceeding. Steps 1–3 prepare credentials and inventory configuration that must be in place before the deployment script runs.

  1. Set up GHCR credentials — obtain a GitHub Personal Access Token with read:packages scope and add it to the inventory as described in GHCR_AUTHENTICATION. If you plan to pre-populate the local Docker registry from GHCR during bootstrap (registry_populate_images: true), credentials must be obtained before Hardware Setup step 7.
  2. Set up SSO — create the OAuth application(s) for your identity provider(s) and add the resulting credentials to the inventory. Before starting, determine your IaaS Console hostname (see Console URL) as it is required for the redirect URI configuration:
    • Google: follow GOOGLE_SSO_SETUP — produces GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET
    • Azure: follow AZURE_SSO_SETUP — produces AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, and AZURE_TENANT_ID
  3. Prepare TLS configuration — create the Azure DNS zone, create a service principal with Contributor access, and add the resulting credentials to the inventory. Also point your public domain to the Azure DNS name servers. Follow steps 1–4 of MANAGEMENT_TLS. Certificate issuance happens automatically during deployment.
  4. Run the full deployment — executes the Ansible playbooks to deploy OpenStack, the management cluster, and the observability stack. The bootstrap and switch configuration steps were already completed during hardware setup, so run only the master script: 
    ./scripts/platform-setup.sh
    See DEPLOYMENT for inventory configuration details, available tags, and troubleshooting.
  5. Complete RADOS Gateway Keystone integration — now that the OpenStack Keystone endpoint is live, complete the post-deployment RADOS Gateway steps: distribute the Keystone CA certificate to all Ceph nodes, configure the Keystone integration settings, and restart the gateway. See steps 4–6 of CEPH_SETUP.
  6. Finalise TLS — once the deployment completes, verify that Let's Encrypt staging certificates have been issued, then switch both iaas_console_tls_cluster_issuer and obs_tls_cluster_issuer to letsencrypt-prod in the inventory and re-run the deployment. See the Deploy section of MANAGEMENT_TLS.
  7. Configure VPN access — add operators to the WireGuard VPN following OPERATOR_VPN_CONFIGURATION.
  8. Verify the system is ready — confirm deployment logs show no failures, management cluster nodes are in Ready state, all pods are Running, and the full product flow works end-to-end. See the Verify Deployment Success section of the deployment guide for the infrastructure checks, and the End-to-End Acceptance Checklist for the full acceptance procedure covering console public access, BGP routing, tenant creation, VPN health, cluster creation, floating IP reachability, object storage, and GPU scheduling.

IaaS Console configuration​

IaaS Console - Tenant and User configuration​

To create additional admin users, register tenants and tenant users, please refer to the instructions in IAAS_CONSOLE_CONFIGURATION.

IaaS Console - Cluster Add-ons​

Cluster Add-ons (such as JupyterHub and KubeRay) are pre-configured in the system. For information on monitoring the feature's health and performing manual maintenance, please refer to the ADDONS_OPERATOR_GUIDE.