OpenStack Ansible Ceph

OpenStack-Ansible (OSA) can integrate with an external Ceph cluster that you manage independently. This approach is best when you already have a Ceph cluster or want full control over Ceph upgrades and maintenance. OSA configures Glance, Cinder, and Nova to use your existing Ceph pools and keyrings.

External vs Managed Ceph

Prerequisites

Step 1: Prepare Ceph Pools and Keys

On your Ceph cluster, create the required pools:

ceph osd pool create volumes 128
ceph osd pool create images 64
ceph osd pool create vms 128
ceph osd pool application enable volumes rbd
ceph osd pool application enable images rbd
ceph osd pool application enable vms rbd

Create service keyrings:

ceph auth get-or-create client.cinder \
  mon 'profile rbd' \
  osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images'

ceph auth get-or-create client.glance \
  mon 'profile rbd' \
  osd 'profile rbd pool=images'

ceph auth get-or-create client.nova \
  mon 'profile rbd' \
  osd 'profile rbd pool=vms'

Step 2: Collect Ceph Details

Gather these values from your Ceph cluster:

ceph fsid                           # cluster UUID
ceph mon dump | grep mon\.          # monitor addresses
ceph auth get client.cinder         # cinder keyring
ceph auth get client.glance         # glance keyring

Step 3: Configure OSA for External Ceph

Edit /etc/openstack_deploy/user_variables.yml:

# Glance with Ceph
glance_default_store: rbd
glance_rbd_store_pool: images
glance_rbd_store_user: glance
glance_ceph_client: glance

# Cinder with Ceph
cinder_backends:
  ceph:
    volume_driver: cinder.volume.drivers.rbd.RBDDriver
    rbd_pool: volumes
    rbd_ceph_conf: /etc/ceph/ceph.conf
    rbd_user: cinder
    rbd_secret_uuid: "{{ cinder_ceph_rbd_secret_uuid }}"
    volume_backend_name: ceph
    report_discard_supported: true

cinder_ceph_rbd_secret_uuid: "457eb676-33da-42ec-9a8c-9293d545c337"

# Nova ephemeral disks on Ceph
nova_libvirt_images_rbd_pool: vms
nova_ceph_client: nova
nova_libvirt_rbd_secret_uuid: "{{ cinder_ceph_rbd_secret_uuid }}"

Step 4: Deploy Ceph Configuration Files

Create /etc/openstack_deploy/user_secrets.yml entries (or use user_variables_overrides.yml):

ceph_mons:
  - 10.0.0.11
  - 10.0.0.12
  - 10.0.0.13

ceph_extra_confs:
  - src: /etc/openstack_deploy/ceph.conf
    dest: /etc/ceph/ceph.conf

Place a ceph.conf on the deployment host at /etc/openstack_deploy/ceph.conf:

[global]
fsid = <your-cluster-fsid>
mon_host = 10.0.0.11,10.0.0.12,10.0.0.13

Place keyring files at /etc/openstack_deploy/ceph.client.cinder.keyring (and similarly for glance and nova).

Step 5: Create the Ceph Client Overrides

Create /etc/openstack_deploy/env.d/ceph.yml to ensure ceph-common is installed in the right containers:

component_skel:
  ceph_client:
    belongs_to:
      - cinder_all
      - glance_all
      - nova_compute

Step 6: Run the Playbooks

cd /opt/openstack-ansible

# If re-running on an existing deployment, target specific services:
openstack-ansible playbooks/os-glance-install.yml
openstack-ansible playbooks/os-cinder-install.yml
openstack-ansible playbooks/os-nova-install.yml

For a fresh deployment, run the full playbooks:

openstack-ansible playbooks/setup-openstack.yml

Step 7: Verify

# Test Glance
openstack image create --disk-format raw --container-format bare \
  --file cirros.img cirros-ceph
rbd -p images ls

# Test Cinder
openstack volume create --size 5 test-ceph-vol
rbd -p volumes ls

# Test Nova (boot from volume)
openstack server create --flavor m1.small \
  --block-device source=image,id=<image-id>,dest=volume,size=10,bootindex=0 \
  --network demo-net test-bfv-vm

Troubleshooting

Summary

Using an external Ceph cluster with OSA gives you the best of both worlds: OSA handles OpenStack deployment and lifecycle, while you retain independent control over Ceph operations, upgrades, and scaling.