Using Puppet with Eucalyptus

Puppet has emerged as a useful continuous configuration management tool. It can not only automate initial install and configuration of software, but also manage the state of machines during their lifecycle. We're hearing a lot of demand for Puppet from our customers and so this post should help people who want to integrate Puppet with their Eucalyptus cloud.

Manage Instances or Machines (or both)

Puppet can be used to help manage machines that make up your Eucalyptus cloud as well as instances running as guests on top of Eucalyptus. I'll cover those separately since they are somewhat different problems (though one Puppet server can service both types of machines).

Automating Configuration of Eucalyptus Infrastructure

Puppet is not used to provision bare metal. For that, you'd rely on PXE, Kickstart or Jumpstart. The machine that is started by one of these methods needs to have the bare OS and minimal packages required for running a puppet agent (client). Since many customers mention they have their own Puppet masters running in their shop, we'll assume that is the case. Another option is to run a Puppet master on the CLC, but with HA enabled, that becomes trickier. Running one Puppet master per CLC, then using Anycast DNS to resolve to a master is recommended by PuppetLabs.

Bare-metal Provisioning

One of several methods (like kickstart, PXE boot, etc.) can be used to install the base OS on new hardware. That base OS must contain a working NTP service and the puppet client services. NTP is required for time-synched requests between the puppet client and master. To integrate with a CLC-based role assignment tool, ssh must also be enabled. The /etc/puppet.conf file must contain the following to allow it to download “fact” definitions from the master;

[main]
pluginsync = true

In addition, an entry should be made in the /etc/hosts file equates the name “puppet” with either a local Puppet master server or a DNS entry that will allow the Puppet master to be resolved. To allow scripted ssh access to the new instance, ssh keys must be generated with the public key being placed in the /root/.ssh/authorized_keys file in the image. The server master (or where the provisioning script is being run) will have the private key of that keypair.

Kicking off Puppet

Once a node is provisioned, it needs to be set up with a role before puppet can take over. The mechanism we're using to tell what type of node this is comes from the contents of the /etc/role file. This file needs to be written (via ssh) prior to the puppet agent running. The best solution is to boot the node, ssh into it and write the /etc/role file, then start the puppet agent and logout. The file would consist of one of the following strings “nc” or “cc”.

Puppet Master Configuration

Puppet allows configuration to be broken into modules. Doing so helps compartmentalize different piece of the config. For this, we've broken this down primarily by service. Each service is simply included in the main site config script which chooses (based on role) which packages to install.

stage { [ "pre", "post" ]: }
Stage['pre'] -> Stage['main'] -> Stage['post']

class first {
  include ntp
  include euca-deps
  include euca-repo
  notify { "myrole":
    message => "my role is $role",
  }
}

class second {
  case $role {
    'cc': { include euca-cc, euca-sc }
    'nc': { include hypervisor, euca-nc }
  }
}

class third {
  include certs
}

node default {
 class { 'first': stage => pre }
 class { 'second': stage => main }
 class { 'third': stage => post }
}

Another thing you'll notice is that there are three stages. This was done to provide some structure since Puppet requires explicit ordering. In other words, just because I put one module after another is not guarantee of their order of execution. Here, we have a bunch of things that must be done (in no particular order) before the eucalyptus packages can be installed. Finally, we'll copy the certs and register the new node.

Setting up the Eucalyptus Puppet scripts requires some amount of configuration that is site-dependent. To enable puppet clients to authenticate with the server, a set of keys is created. Normally when the client starts, it needs to request keys and the master needs to authorize the client. On a trusted network, it might be desirable to auto-sign new clients, so the autosign.conf file would need to be configured with an IP range where new servers will come up. Eucalyptus keys need to be shared with new nodes after the package installation is done. To do this, a puppet file server is configured with the fileserver.conf file. That file contains the path to the keys (which varies depending on package or src install of eucalyptus) and the CIDR to allow access to the files.

Once the new node has been installed, a normal install would have you run euca_conf to register the component(s). We'll need to do that as well, but this time specifying the –no-rsync option. That is because rsync is used to copy keys which we're doing with puppet now.

Managing Guest Instances

Managing software configuration on instances running in the cloud is a challenge and many different solutions are available. Some people simply roll new images and boot replacement instances while deprecating the old ones. Others moved up to scripts that can update the software, often upon reboot of an instance. There are many vendors offering management tools that let you define the software stack and automate rollout of changes. There is another class of solutions that can manage live updates and configuration changes. Puppet falls into that category and it seems to be very popular with our customers.

Central to the way Puppet works are scripts that they call manifests. They describe resources that have a set of attributes which are monitored and kept in synch with a server. The server is the Puppet Master. Many Puppet users manage their own master server, so there may not be a need to set up a new one. People who have never used Puppet would need to dedicate a server (or space on a server) to running the master service and housing the manifests.

While there are different mechanisms, machines managed by puppet are configured based on some criteria (role, IP range, hostname, etc.). Puppet provides an abstraction layer over many of the resources it manages. In some cases, you need to (and are allowed to) specify differences based on the OS type (or other attributes).

Using Puppet for instance configuration management requires a Puppet Agent to be running on the instance. One method is to have puppet installed and configured to run as part of the base OS on the image. Another would be to use cloud-init to install and launch the puppet agent at boot time. In either case, the server would eventually run the Agent software which contacts the local Puppet Master. When that happens, the master determines what is required for that machine and replies with a catalog of configuration changes that must be applied to that machine. Beyond the initial configuration, Puppet also manages configuration changes during the instance lifecycle by contacting the master periodically and applying any change necessary to ensure proper configuration.

Configuring a Base Client Image

When preparing for the first Puppet/Eucalyptus webinar, we struck a balance between installing software on the base image and leaving some configuration for boot time. The resulting image used a base (in this case, CentOS) image and added the Puppet Enterprise download. This was un-tarred and left ready to install. There is a way to run the puppet-enterprise-installer (using the -s option) to save the answers to install questions to a file and not actually run the install. In this way, it is possible to pre-answer all of the install questions to enabled a scripted install at boot time. Two of the questions require input that is best provided when the script is run, vs when the image was created. These things are the IP (or name) of the puppet master, and the name of the agent node. For the demo, we used IP address, but you may wish to use a DNS name to have something more meaningful vs. just an IP. Here's an example of the first-run script used to install puppet a boot time.

CERTNAME=`curl http://169.254.169.254/latest/meta-data/local-ipv4`
PUPPETMASTER=`curl http://169.254.169.254/latest/user-data/`

cat <<EOF> /tmp/pe-client-install
q_install='y'
q_puppet_symlinks_install='y'
q_puppetagent_certname=$CERTNAME
q_puppetagent_install='y'
q_puppetagent_pluginsync='y'
q_puppetagent_server=$PUPPETMASTER
q_puppetdashboard_install='n'
q_puppetmaster_install='n'
q_rubydevelopment_install='n'
q_vendor_packages_install='y'
EOF

/root/puppet-enterprise-1.0-centos-5-x86_64/puppet-enterprise-installer -a /tmp/pe-client-install

Notice that instance metadata is used to get the local IP address. User data is used for the puppet master address, which means when this instance is launched, the puppet master IP address needs to be passed as user data. Once this script runs, the puppet agent will be running and communicates with the certificate authority on the master. At this point, it is up to the administrator to authorize this server (by issuing a “puppet cert -s <agent-ip-address>” command). Once the certificate is signed, the agent will start communicating with the master in a normal way to download a catalog of changes that need to be made. When using Puppet Enterprise, the dashboard can be used to assign a role (class) to the node. The class (or classes) determine the packages, files, etc. that need to be installed or modified. For completeness, add the following to the ”/etc/rc.local” file, assuming the above script is stored in ”/etc/rc.first”.

if [ -e /etc/rc.first ]
then
  /etc/rc.first
  mv /etc/rc.first /etc/rc.first.done
fi

The image used in the webinar is available on the Eucalyptus Community Cloud as emi-517C1318.

Master Configuration

Installation of the puppet master is straight forward. After starting a base OS image, simply use “wget” to download the appropriate puppet release. Once the bundle is un-tarred, run the puppet-enterprise-installer script and answer the questions. Once the installation is done, the master and dashboard services will be running. The dashboard responds on port 3000 by default, so that needs to be open in the security group that server is running in. Your own puppet modules should be placed in /etc/puppetlabs/puppet/modules. The dashboard will allow you to associate classes with nodes. The ECC contains an image (emi-F7251183) that runs Puppet Enterprise, configured with it's IP as the certificate name.

Attachment	Size
PuppetIntegrationGuide.pdf	221.51 KB