OpenStack Profile Updated to Mitaka and Ubuntu 16.04

We've done another round of updates the OpenStack profile and the images it's based on, and wanted to share the important changes -- and encourage you to migrate to this latest version insofar as possible.  The current version will remain available for instantiation for a few weeks, but then we will disable it.  Here's a quick summary of the new features:

• Mitaka is now the default OpenStack release configured by this profile.  Kilo and Juno are deprecated.  We are no longer testing the profile's functionality under those versions, and we plan to remove these version options in the next version of the profile.  You should update to Mitaka if possible.
• The default topology now includes only two nodes: a controller ("ctl") node and a compute ("cp-1") node; the networkmanager node's functionality has been moved to the controller, as is the default in the OpenStack Ubuntu/Apt documentation.  You can return to the old three-node configuration by changing the name of the networkmanager node in the Advanced Parameters from "ctl" to "nm".
• We have added support for the Neutron ML2 "linuxbridge" driver, although we continue to configure the "OpenVSwitch" ML2 driver by default.  You can select this option by parameter; although if you do, you must also set the number of GRE tunnels to 0, and you will probably want to set the number of VXLAN tunnels to 1 (the "linuxbridge" driver does not support GRE tunnels).  Although OpenStack has switched to the linuxbridge driver as its default, we have no plans to do that, since OpenVSwitch is quite useful to researchers.
• One of the interesting new Mitaka features is shared filesystem support (the "manila" component).  We download a shared filesystem image and configure Manila so that you can immediately create a share and connect it to guests.
• We have added the ability to specify extra images to be included in your OpenStack.  So if the supplied trusty-server image doesn't provide what you need, you can drag in other cloud images by filling in the "Extra Image URLs" parameter.  For instance, you could specify "https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img https://download.fedoraproject.org/pub/fedora/linux/releases/24/CloudImages/x86_64/images/Fedora-Cloud-Base-24-1.2.x86_64.qcow2" and this would get you the default trusty-server image; the latest Ubuntu Xenial Xerus (16.04) image; and the Fedora 24 Cloud Base image.  You could customize the image names if you wanted, by appending a name after an ":" after each URL (i.e. "https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img:XENIAL-SERVER https://download.fedoraproject.org/pub/fedora/linux/releases/24/CloudImages/x86_64/images/Fedora-Cloud-Base-24-1.2.x86_64.qcow2:FEDORA-BASE").
• You can add more images after boot.  Become root, "cd /root/setup", and run "bin/import-image.sh <URL>[:image-name]".  The image import script supports MBR single-partition images only.  It supports compressed images (gz, xz); and these image formats: qcow (any version, theoretically), vmdk, vhd, and raw disk images.  If you have the option, it is best to stick with qcow.  It supports Ubuntu, Centos, and Fedora cloud images.  The import script forces the random passwd your profile was assigned into /etc/shadow for the root and the ubuntu, centos, or fedora user account (each of those user names is the canonical cloud user id for the respective distribution).  This helps ensure your image won't boot with an unsafe passwd.
• The image import script makes some modifications to automatically support multiple NICs (and thus there is no more "trusty-server-multi-nic" image in the image chooser).  By default, cloud images support only a single NIC.  We have added some scripting in the boot path to automatically DHCP on all Ethernet devices.  If you want to use this feature, you just need to pay attention to which interface you bind a floating IP to, because only one interface is associated with the VM's default route.  The DHCP client gets DNS/route/NTP information for each interface from OpenStack; but this information is discarded for the non-eth0 devices.  This means that the first network you add to your instance when creating it is the interface you must bind your floating IP to, because the default route will be associated with that interface (eth0).
• On x86_64, the image import scheme supports only single-partition images, not referential AMI/AKI/ARI tuple images.  You can't currently specify those.  However, on aarch64, we only create AMI/AKI/ARI tuples.  We take single-partition images and convert them into AMI/AKI/ARI tuples by pulling out vmlinuz and initrd from /boot in the single-partition image.
• In the past, the trusty-server* images on aarch64 were pieced together manually.  We no longer bother with that complexity.  Instead, we download the Ubuntu trusty-server disk1 img, and modify it to support our needs, as described just above (image import for aarch64).  Thus, the aarch64 image may feel slightly different.
• The MTU that dnsmasq pushes to your OpenStack VMs via DHCP as they boot has been reduced from 1454 bytes to 1450 bytes.  1454 is an adequate setting for GRE tunnels, of course, but not for VXLAN networks, which require 1450 bytes on a normal physical network with 1500-byte MTU.  Somehow this mistake escaped prior testing.
• You can now specify an Apt mirror and set a custom mirror path if you require fast localized access to a mirror.

Thanks for reading, and please report any problems to cloudlab-users@googlegroups.com .  If you're not a member, please join!

Ubuntu 16.04 Standard Images

We've released UBUNTU16-64-STD (x86_64) and UBUNTU16-64-ARM (aarch64), new standard CloudLab disk images with Ubuntu 16.04 LTS (Xenial Xerus) 64-bit. The images contain a 4.4.0 Linux kernel, gcc 5.4, systemd 229, etc. All Emulab services are launched and controlled via udev and/or systemd.  You can find more information at https://wiki.ubuntu.com/XenialXerus/ReleaseNotes . Please test it out, and let us know at support@cloudlab.us if you find anything broken, or generally useful packages that are missing.  You can refer to the image names or URNs listed below in your profiles.

    UBUNTU16-64-STD: urn:publicid:IDN+emulab.net+image+emulab-ops:UBUNTU16-64-STD
    UBUNTU16-64-ARM: urn:publicid:IDN+utah.cloudlab.us+image+emulab-ops:UBUNTU16-64-ARM

OpenStack Profile "Liberty" Support

We've done another round of updates the OpenStack profile and the images it's based on, and wanted to share the important changes -- and encourage you to migrate to this latest version insofar as possible.  Here's a quick summary:

• Liberty support (Kilo and Juno still available, but upgrade if you can)
• Keystone v3 API enabled by default for both Kilo and Liberty (but can select v2.0 if preferred)
• Migrate (for Kilo and greater) to the "openstack" CLI client for configuration, instead of the per-service CLI clients
• Parameters for choosing node type and link bandwidth
• Increase token and horizon (dashboard) timeouts to let web users remain logged in longer (these are parameters with long default values)
• Migrate (for Kilo and greater) to Keystone via WSGI/Apache (but this is also a parameter, so you can select the old method of the Keystone Python API server)

We've traditionally configured OpenStack in accordance with the installation documentation, using their defaults when possible.  However, this time, there are some notable changes:

• Keystone doesn't use Memcache by default (although it's an option)
• We continue to use the openvswitch Neutron driver to manage networks; the Liberty docs have switched to the linuxbridge driver
• We continue to use a split controller/networkmanager installation, unlike the docs, which now unite the controller and networkmanager.  We'll probably migrate to this eventually.
• We set the default resource limits to unlimited for Nova, Neutron, and Cinder (the default resource limits can be left intact by unchecking the quotas parameter)

Thanks for reading, and please report any problems to cloudlab-users@googlegroups.com .  If you're not a member, please join!

Glibc Vulnerability Patching

Hi all,

In order to apply patches for the recent glibc resolver buffer
overflow vulnerability, we plan to reboot all of the CloudLab control
servers today at 5PM MST. This will temporarily interrupt
instantiation of new experiments, and the CloudLab web portal will
also be unavailable for 15 minutes or so.

Related to this glibc vulnerability, we ask that you:

* Please perform a software update on nodes in running experiments

If you expect that your experiment(s) will run for more than two days
from now, please update your nodes via the running OS's distribution's
update mechanism:

As root on Ubuntu:

apt-get update
apt-get upgrade
reboot

As root on CentOS:

yum update
reboot

Notes: If "grub" is updated in this process, it may ask where it
should install itself.  Choose "/dev/sda1" for anything other than
Ubuntu 12.  For Ubuntu 12, choose "/dev/sda2".  Also choose to keep
any existing configuration files if/when prompted (e.g., for Grub,
OpenSSH server, etc.)

* Please update your custom disk images

If you use a custom disk image, please perform a system software
update as described above, and re-snapshot your image.

Email support@cloudlab.us with questions.

More info on the glibc vulnerability can be found here:

https://access.redhat.com/articles/2161461

OpenStack Profile and Image Changes and Updates

We've updated both the OpenStack profile and the images it's based on, and wanted to point out some important changes -- and encourage you to migrate to this latest version insofar as possible.

First, we've changed the way OpenStack packages are installed and/or upgraded by the scripts.  Originally, for several reasons, we wrote the profile's setup scripts to always install the latest version of the OpenStack packages.  That no longer makes sense, because 1) the Ubuntu OpenStack packages are pretty stable right now; and 2) we can (and now do) provide a profile parameter that allows you to upgrade to the latest packages if you like.  However, by not updating the pre-installed packages by default, we can significantly reduce the load the scripts put on the Ubuntu package mirrors -- and more importantly, provide a more stable profile that isn't a moving target.  This more reasonable default is long overdue; thanks for waiting for it.  There are now three options that affect package installation on the nodes in your OpenStack experiments.  The first is new; the latter two were previously present.

• Upgrade OpenStack packages and dependencies to the latest versions
  If a package is already installed, we don't try to upgrade it to the latest version unless this option is selected.  The default is false (unselected).
• Install required OpenStack packages and dependencies
  If this option is false (unselected), the setup scripts assume all required OpenStack packages are installed, and it only installs critical dependencies that it absolutely requires for its own execution.  By default, of course, this option is true (selected).  If the "Upgrade" option detailed above is false, and OpenStack packages are already installed, nothing will happen.  If packages are not pre-installed, if this option is selected, they will be installed.
• Update the Apt package cache before installing any packages
  This option gives you control over whether or not the scripts update the Apt package cache before doing any package installation/upgrades.  Typically, it's a bad idea to not update the cache, as you may end up trying to install packages that are no longer present on the mirrors; but this gives you that choice just in case.

Second, we've updated the Cloudlab software installed on the images the profile uses, to pull in some important updates.  When you pick Kilo as your OpenStack version, the Ubuntu 15 images your experiment will use now support swap partitions on x86 (most Cloudlab images have a swap partition in the standard partition layout, of course, but the right systemd helper services were not installed on Ubuntu 15).  The ARM-specific images (usable only at Cloudlab Utah, our ARM-based cluster) use our more modern partition layout, so instead of a whole-disk root partition, there is space for you to create partitions.  In particular, the setup scripts create a secondary partition as a backing store for an LVM physical volume.  The ARM-specific images now also have an important Cloudlab software update that allows the system boot initramfses to be properly regenerated (important if the kernel is upgraded, or if the installation of some package triggers the automatic rebuild of the initramfs).  Finally, all images have the most recent Ubuntu package versions of OpenStack Kilo and Juno pre-installed (unless you select the option to start "from scratch", which deliberately uses images that don't have the packages pre-installed).

Third, we've improved the profile's documentation a little bit.  When you swap in an experiment, you'll see a markdown rendering instead of the giant glob of text.  Hopefully this will make things more clear -- although we didn't attempt to document everything.

Finally, we've disabled non-current versions of this profile, meaning that you cannot instantiate nor copy those profile versions.  Please instantiate using the latest version.

Thanks for reading, and please report any problems to cloudlab-users@googlegroups.com (if you're not a member, you should join!).

Changes to the default OpenStack profile

We have three announcements to make today regarding the default OpenStack profile in CloudLab:

1. The OpenStack profile now randomly generates a new password for every experiment. This password is used for the 'admin' login in the OpenStack web interface and for password ssh logins to VMs created by OpenStack. The password for your experiment can be found in the "Instructions" panel in the experiment status page. 

   

   OpenStack profile instructions showing admin/root password

2. It's now easier to tell when OpenStack is done setting up. The Topology view of the experiment status now has little icons on each node showing the status of the scripts that set up that node. Now, when OpenStack setup is complete, all of these icons will change to checkmarks, and the 'State' of the experiment will change from "booted" to "ready". It's normal for the control node to take much longer to finish setting up than the compute or network manager nodes; it has a lot more work to do.



Profile topology view showing three ready nodes

3. We are deprecating the "Tutorial-OpenStack" profile in favor of the "OpenStack" profile. The "OpenStack" profile covers all features offered by the tutorial version, and more. We are not deleting the Tutorial-OpenStack profile at this time, but it is no longer selected by default, and we do not encourage people to use it for new experiments.

Important Tutorial-OpenStack and OpenStack Profile Change

For security reasons, we have changed the CloudLab-provided OpenStack profiles to modify the way root login is handled on the VMs brought up by OpenStack.  This also affects most other OpenStack profiles on CloudLab (those that use our OpenStack setup scripts). Note that this change does not affect CloudLab profiles that do not use OpenStack.

Password login is no longer allowed for the "root" account. If you need to log in directly to your VMs as root, you will need to use an ssh keypair. Password login is still allowed for the 'ubuntu' account, so if you do not have a keypair set up, you may use that account instead.