After the philosophical first part, now is time to get to work. Lets see how to use Ironic to show how beneficial is managing resources using OpenStack API’s, starting from zero using Ansible to create a standalone Metal as a Service.
Metal as a Service implementation with Ansible
- All clients are allowed to enroll/deploy/decommission servers. The user just need to know the URL of the Ironic-Api and provide an empty (fake) token (no Keystone integration).
- The network will be plain, at this time, there is no need of controlling switches and ports (no Neutron integration). The Ironic server and bare metal clients are connected directly to the PXE network (default native vlan) using one NIC. The same applies for the IPMI network, it is possible to setup another NIC to specifically access that network.
- Images are saved on the Ironic server, on local disk, and are available to other servers by HTTP when using Ironic Python Agent driver (IPA) or by TFTP when using PXE with the default driver (by Ironic-Conductor). This is not needed if you want to use other http servers or even object storage (like S3 or swift).
The server deployment was tested on Ubuntu Trusty, and the setup playbook will install all packages from the distribution repositories and the official Ubuntu Cloud Archive repository, so no development packages are installed, just official packages needed for running.
The ansible roles have no dependecies between each other, so it can be reused in other projects, they have more parameters than those required for this setup:
roles/mysqlto setup a MySQL server and databases needed for Ironic services.
roles/rabbitmqto setup a message queue using RabbitMQ.
roles/ironicto setup the OpenStack Ironic daemons and services.
roles/dnsmasqto setup a PXE server to use with Ironic conductor.
roles/configdrive(only for the client playbooks)
roles/monit(optional) to setup process control with Monit.
roles/nginx(optional) to setup HTTP image repo server (for IPA).
The last two roles are not really needed if you do not want to use Monit and if you
have another HTTP storage repository where images can be saved and reachable
from the bare metal servers. Note that the
configdrive role is only needed for the
client playbooks (
del-baremetal.yml), not for Ironic
server, and it will also make use of the HTTP server, so as starting
point, probably it is better just using it, after that you can start with your
modifications. The roles were implemented following
these patterns, trying to
make them simple and reusable, but the idea can be implemented in other
provisioning tools easily.
Installing Ironic server
setup-ironic.yml calls the previous roles with the proper parameters.
The deployment can be splitted on different servers: database, messaging, api,
conductor by defining the hosts/ips on the inventory file
Also, in that inventory file, there are the main variables used in the playbooks,
like the dhcp range for PXE and the passwords for MySQL and RabbitMQ.
baremetal_server_dhcp_params defines a range for the ramdisk, but note that it
could be completely different than the final IP address that the bare metal host
will have (because it can be defined by Cloud-Init), for this reason, the lease
time can be small (~30m) and the IP range can be short (~20 ips).
[provisioning:children] database messaging ironic client api conductor [provisioning:vars] # Server side variables baremetal_server_database_rootpass=root baremetal_server_database_name=ironic baremetal_server_database_user=ironic baremetal_server_database_pass=mysql baremetal_server_msg_vhost=ironic baremetal_server_msg_user=ironic baremetal_server_msg_pass=rabbitmq baremetal_server_dhcp_params=["10.0.1.100", "10.0.1.120", "30m"] baremetal_server_dhcp_domain=pxe.local baremetal_server_http_root_path=/var/lib/ironic/http baremetal_server_images_path=/var/lib/ironic/http/images baremetal_server_images_deploy_path=/var/lib/ironic/http/deploy baremetal_server_configdrive_path="/var/lib/ironic/http/metadata" # Client variables baremetal_ironic_token=" " baremetal_domain=example.com baremetal_nameservers=[ "220.127.116.11" ] baremetal_ssh_key="ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB...tion"
The current implementation installs all the services on the same host. Apart
of the MySQL and RabbitMQ services, it is possible to split the Ironic-API and
Ironic-Conductor in different servers (or deploy multiple Ironic-conductor’s)
toggling the parameters
ironic-api on the ironic role:
- hosts: api sudo: true roles: - role: ironic ironic_api: true ironic_conductor: false # ... - hosts: conductor sudo: true roles: - role: ironic ironic_api: false ironic_conductor: true # ... - role: dnsmasq
Note that each Ironic-Conductor host also needs the
dnsmasq role. So, on the
ansible inventory file (
hosts/ironic in this case), remove
conductor from provisioning section, and add:
[ironic:children] api conductor
It is possible to have and HTTP image repository on each Conductor by adding
nginx role, but then you have to be aware of distributing the images
to all Ironic-Conductor’s servers.
To run the playbook, just change the IP’s and ssh user in the inventory file to
your environment. Note that Ansible thinks that is deploying on different hosts
(the reason why there are different host names pointing to the same IP is to
avoid overwriting the Monit control scripts), but you can change and
simplify that configuration by changing the hosts definition of the playbooks
according to your inventory file (have a look at vagrant playbook
Finally, remember that the playbook is idempotent, so run it as many times as
you need, it will not restart the services unless is necessary:
ansible-playbook -i hosts/ironic -e setup-ironic.yml
Ironic Python Agent ramdisk
IPA is a an agent for controlling and deploying Ironic controlled bare metal machines running in their memory:
- The agent is a small python application that is meant to be embedded in a ramdisk.
- The target machine boots this ramdisk, which starts the agent process.
- The agent then exposes a REST API that Ironic uses to interact with the agent.
- The API calls pluggable backends to perform actions on the machine.
One of those actions is deploy a image on the physical machine, others could be wiping the disk, create RAID setup …
The IPA ramdisk image must the copied to Ironic-Conductor(s) because is used by
Dnsmasq to offer it to the unprovisioned machines via PXE. The agent_ipmitool
driver on Conductor will power the machine and control the boot order (PXE vs disk).
The playbook to install the Ironic server
setup-ironic.yml already does it for you …
- name: Download IPA Coreos ramdisk get_url: url: http://tarballs.openstack.org/ironic-python-agent/coreos/files/coreos_production_pxe_image-oem.cpio.gz dest: "/coreos_production_pxe_image-oem.cpio.gz" force: no - name: Download IPA Coreos kernel get_url: url: http://tarballs.openstack.org/ironic-python-agent/coreos/files/coreos_production_pxe.vmlinuz dest: "/coreos_production_pxe.vmlinuz" force: no
… but attention, those tasks will run only once, when Ironic is updated they will not update the IPA ramdisk, to force the update, just rename those files and re-run again to download the new version (or do it manually). Also, if you install the Ironic Kilo version (or any version different that the latest), the IPA you will get is not going to be compatible it, because it always downloads the latest release.
Take into account, when defining a new physical machine with Ironic client, one have to specify the path to IPA ramdisk kernel and image relative to Ironic-Conductor service. This is not the ideal situation, because a client should not know those details, but for now, the ugliness can be mitigated by the playbooks, hidden those details to the end users.
Creating images to deploy on the physical machines
The last step needed to start using our Metal as a Service is create the images to deploy on the physical machines. Because of we are using IPA ramdisk to deploy the image, it is needed some requirements to be compatible with this method of deploy:
- Whole disk image with the boot loader included.
- If not using GPT, the image cannot have more than 3 primary partitions.
- Cloud-Init running at boot time with config-drive support.
- RAW or qcow2 image format.
The second point is because the IPA will create one partition for the config-drive after the image is deployed, if there are no free primary partitions (the maximum is 4), it will not be able to do that and the process will fail.
In order to automate the process I have decided to use Packer instead of the official disk image tools, because it allows to define support for LVM on hardware for managing snapshots, resizing disks, physical disk independency to move filesystems, etc. Also, I would like to define a simple way to add more features (drivers, programs, configurations, … ) using scripts, and packer makes it easy to implement.
The repository has some examples about how to create images, let’s see how to install and run packer to create an Ubuntu Trusty image:
# Download packer (choose the latest version) wget https://dl.bintray.com/mitchellh/packer/packer_0.8.6_linux_amd64.zip -O /tmp/packer.zip sudo unzip /tmp/packer.zip -d /usr/local/bin rm -f /tmp/packer.zip
Install Qemu-kvm to run a local VM and create the template
# Install the packages sudo apt-get install qemu-utils virtinst virt-viewer libguestfs-tools sudo apt-get install qemu-kvm qemu-system libvirt-bin bridge-utils virt-manager # Your user should be part of the libvirtd group to manage vms sudo adduser $USER libvirtd sudo adduser $USER kvm # change the current group ID during this session. newgrp libvirtd newgrp kvm # Check if everything is working: virsh -c qemu:///system list
Run Packer using a template definition. Have a look at the pressed file on the folder
ubuntu/httpto define an alternative disk layout on the image. The rest of the configuration settings embedded on the image are defined by the scripts on the folder
# Building an ubuntu baremetal image cd ubuntu packer build ubuntu-14.04.latest-amd64.json # output image in the new folder output-ubuntu-14.04
Finally, just copy the image to the HTTP repository -Nginx, in this implementation, the same server as Ironic-Conductor(s)- to be ready to use and workout the md5 checksum
md5sum output-ubuntu-14.04/trusty.qcow2 > output-ubuntu-14.04/trusty.md5 # Copy the md5 and qcow to the location defined by baremetal_server_images_path # (on the ironic role is ironic_pxe_images_path) scp output-ubuntu-14.04/trusty.* <IRONIC_SERVER>:/var/lib/ironic/http/images/
The image has to be available on http://<IRONIC_SERVER>/images
Deploying a physical machine
Once the image is on the Ironic server, it is time to use the client playbooks to deploy a physical server. Of course, you can use directly the Ironic command line, but it will be a bit difficult to manage with a lot of servers (also for creating the Cloud-config configuration).
Those playbooks rely on the Ironic client, but in the future with ansible 2.0 they will use the openstack os_ironic and os_ironic_node modules to simplify the tasks. Anyway, they are good examples to follow the steps needed in the manual process.
Also note that now is time to use the role
configdrive, which is mainly in
- Creating the network configuration files on the nodes, it is capable of creating complex network configuration: bonding, vlans, … with static IP’s. When the server boots for first time, it will do it with the proper network configuration. It works on RedHat and Debian derivatives distributions.
- Provide the
user_datascript for Cloud-Init, which is in charge of creating additional users, extend the LVM and the filesystems, trigger other hookups, etc.
- Create the ISO volume with all the configuration files according to the format needed by Cloud-Init.
Those actions are executed on the HTTP (Nginx) repository, due to the fact that
the images and the config-drive specified when the physical server is defined,
will be picked up by the IPA agent from there in order to be dumped on the
physical disk. The config-drive configuration will be generated in two formats:
a folder named with the
<UUID> of the physical machine (useful only for the
humans, it can be deleted -see configuration parameters of the role-), and the
compressed ISO volume of the same folder called
<UUID>.gz. Remember you can
check them in [http://
Before deploying the machine, have a look at the file
check and, if it is needed, re-define these variables:
baremetal_conductor_server: "" baremetal_ironic_url: "http://:6385"
Depending on the inventory file, you will have to point to the hostname (or IP address) of the Ironic-Conductor and setup the Ironic-API url (in this configuration both are pointing to the same server).
Also, is good to have a look at some settings with regards to
the image you want to deploy. Edit, copy or create a file
in this case
vars/baremetal_image-trusty.yml (we have created an Ubuntu Trusty image).
--- baremetal_driver: "agent_ipmitool" # Folder relative to the servers baremetal_deploy_kernel: "file:///coreos_production_pxe.vmlinuz" baremetal_deploy_ramdisk: "file:///coreos_production_pxe_image-oem.cpio.gz" # Depends on each image baremetal_image_type: "Debian" baremetal_image: "http:///images/trusty.qcow2" baremetal_image_checksum: baremetal_image_kernel: baremetal_image_ramdisk: baremetal_image_rootsize:
The image checksum can be empty if you have created the md5 file
next to the qcow2 image on the HTTP repository. The rest of the parameters are
not needed because it is a complete disk image, deployed by the
Now, finally, let’s deploy a machine! Create a file with the parameters of the server
servers folder, for example
--- # Specific variables for this server baremetal_ipmi_ip: 10.0.0.1 baremetal_ipmi_user: IPMIUSER baremetal_ipmi_pass: IPMIPASS # Main network parameters, this variable is needed for the playbook. # It can be in a different VLAN or network than the PXE one baremetal_fqdn: test-server-01.example.com baremetal_mac: 00:92:fa:ab:0e:ba baremetal_ip: 10.100.100.10 baremetal_netmask: 255.255.255.0 baremetal_gw: 10.100.100.1 # Image name, defined in vars/baremetal_image-<name>.yml baremetal_os: trusty # If you change the interface name to eth1 (for example) # Remember to change the backend devices! baremetal_network_list: - device: "bond0" type: "bond" bond_mode: "1" address: "" netmask: "" gateway: "" nameservers: ["18.104.22.168"] domain: "" backend: ["eth0", "eth2"] - device: "eth1.500" type: "vlan" address: "192.168.100.100" netmask: "255.255.255.0" backend: ["eth1"] # Remeber, the image needs support for bonding, vlan, bridge ... # in order to get then working.
Optionally, with the same name but different extension define a
for Cloud-Init. In this case it will create an user and expand the LV and FS
to fill all the available space:
#cloud-config users: - name: admin lock_passwd: False plain_text_passwd: 'admin' gecos: Admin User groups: [wheel, adm, audio, cdrom, dialout, floppy, video, dip] sudo: ["ALL=(ALL) NOPASSWD:ALL"] shell: /bin/bash runcmd: - lvextend -r -l +30%FREE /dev/mapper/system-root - lvextend -r -l +60%FREE /dev/mapper/system-var - lvextend -r -l +10%FREE /dev/mapper/system-tmp
And run the ansible playbook, providing the name of the server/file:
ansible-playbook -i hosts/ironic -e id=test-server-01 add-baremetal.yml Server Name: test-server-01 PLAY [Define and deploy physical servers using Ironic] ************************ [...]
and done! just check the Ironic client to see the status:
export OS_AUTH_TOKEN=" " export IRONIC_URL=http://<IRONIC_SERVER>:6385/ ironic node-list