Tag: ubuntu

OpenStack in action 2! – Production ready / Paris, 31 May 2012

June 5th, 2012 — 10:30pm

Last Thursday, we held the «OpenStack in action 2 ! – Production ready» event in Paris. For this second edition, we were very happy to welcome more than 150 people who came from France, Spain, the UK, the Netherlands, Austria, and even…Japan ! This shows how OpenStack is getting bigger and more concrete every day.

Following an «almost traditional» opening by Thierry Carrez, Release Manager for OpenStack, Loic Dachary, eNovance Chief Research officer, presented a very intuitive OpenStack live demo. The idea was to make an online game go worldwide through a deployment demo. The game is very cool to play by the way, you can give it a try here: MIOP game.

Nick Barcet, representing Ubuntu/Canonical, who is also a regular guest, talked about MAAS and Juju to deploy an Ubuntu Cloud infrastructure. Nick also announced our recent partnership : eNovance is now the official Canonical partner to deliver OpenStack based solutions in France!

Here at eNovance, this 2nd edition of «OpenStack in action!» will remain a very important date for us since we launched the same day eNocloud – the 1st European OpenStack cloud, which was in beta for the last 9 months. Raphaël Ferreira, eNovance’s CEO explained how eNocloud was built, step by step, and related the use cases of our most famous beta-testers: Soasta, Fasterize and the CNES.

After a short coffee break, Taco Scargo, from Dell talked about Crowbar and announced the release of Copper – an experimental yet very promising ARM server.

He was followed by Rick Clark who highlighted the most challenging problems OpenStack currently faces. His perspective as a co-founder of the project and his successful deployment of OpenStack for the US Department of Defense for CISCO gave the audience an exclusive peak at one of the largest use case in the world. His unbiased and provocative style made for a chaotic and humorous talk.

Florian Haas, CEO of Hastexo, closed the conference and really captured his audience whis his talk on High Availability in OpenStack. He explained how the Folsom release that will be published in october 2012 will benefit from the various HA strategies demonstrated.

We wanted to address our special thanks to Canonical and Dell for helping us make the event possible: we look forward to the 3rd edition of «OpenStack in action!» that will be held this autumn.

Comment » | community, Event, Meetup

Automating Openstack Testing on Ubuntu

February 8th, 2012 — 8:44am

(Original Post)

During the Ubuntu precise development cycle the Canonical Platform Server Team have been working on automating testing of Openstack on Ubuntu.

The scope of this work was:

  1. Per-commit testing of Openstack trunk to evaluate the current state of the upstream codebase in-conjunction with the current packaging in Ubuntu precise and the current Juju charms to deploy Openstack.
  2. SRU testing for Openstack Diablo on Ubuntu 11.10.

Openstack do a lot of pre-commit testing through the use of gerrit with Jenkins; we wanted to supplement this with Ubuntu focused testing to provide another dimension to the testing already completed upstream.

So grab a coffee and make yourself comfortable; this is not a short read….

Lab Setup

The Ubuntu Openstack QA lab consists of 12 servers; the primary server in the solution is an Ubuntu 11.10 install providing the following functions:

  1. Juju – used to deploy Openstack charms in the Lab
  2. Cobbler to support server provisioning (using the Ubuntu Orchestra packages in Oneiric)
  3. Jenkins CI – provides triggering based on upstream commits to github repositories and general job control and reporting.
  4. Schroots for Oneiric and Precise for building packages locally
  5. A reprepro managed local archive for Oneiric and Precise
  6. Squid based archive caching to reduce installation times in the lab

This server also acts at the gateway into and out of the Lab (it’s setup as a NAT router).

The other 11 servers are registered in Cobbler; All servers are connected to a Sentry CDU (Cabinet Distribution Unit) which allows full power control from Cobbler – thanks goes to Andres Rodriguez for developing the required fence component for Cobbler to support this type of CDU.

Preseeded LVM Snapshot Installs

To initiate a new integration test run requires all machines to be powered down and re-provisioned from scratch.  It is essential that our deployment and test runs can cope the frequency of upstream commits, particularly as the frequency increases as Openstack approaches milestones and releases.   After getting the initial lab setup in place, we were able to tear down all machines, re-provision and deploy Openstack in ~30mins.

It was important that we are able to minimize the time taken to complete the testing cycle.   To do so, we’ve employed the use of LVM snapshotting and restoration of the root partition during the the netboot installation.   The process is as follows:

  1. Test run begins
  2. Juju deploys a service (i.e. nova-compute)
  3. A machine is netbooted and a preseeded LVM-based Ubuntu installation takes place onto /dev/qalab/root
  4. At the end of the installation, the root filesystem is moved to /dev/qalab/pristine-[release]-root and a snapshot created at /dev/qalab/root
  5. The machine reboots, runs Juju and deploys nova-compute as pat of the rest of the Openstack deployment. This deployment is smoke tested.
  6. The next test run begins.  All machines are terminated. Juju redeploys nova-compute, a machine is netbooted and Ubuntu installation kicks off.
  7. The installation checks for the existence of a logical volume at /dev/qalab/pristine-[release]-root.  If it exists, it creates a new snapshot at /dev/qalab/root and reboots. If it does not, continues with installation and goto step 4.
  8. System reboots, Juju installs and redeploys nova-compute to a fresh Ubuntu installation.

This process takes place on all nodes in parallel.  With it in place, we were able to cut down the time it took to tear-down and re-provision a node from ~30 minutes to 10 to 15 minutes depending on the service being deployed.

By taking this approach we are also minimize the chance of any nodes hitting an archive inconsistency during installation. This is a known issue when deploying the development release and halts installation on any node that hits it, failing the entire deployment.

All of this is embedded in debian-installer preseeds via Cobbler snippets.  The snippets and kick starts are available at lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot.

In the future, we’ll be investigating the use of kexec as an alternative to reboot after snapshot restoration to reduce the time spent waiting on servers to boot.  This should minimize the test cycle even more. Credit to James Blair for the idea (see http://amo-probos.org/post/11).

Management of Jenkins

All of the projects in Jenkins are managed using Jinja2 XML templates in-conjunction with python-jenkins (python-jenkins); this makes it really easy to setup new jobs in the lab and reconfigure existing ones as required (as well as providing great backup!).

Templates and management scripts can be found in lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Testing Openstack Essex on Ubuntu Precise

This testing was the first to be setup in the lab.  Jenkins (using the git plugin) monitors the upstream github.com repositories for commits on the master branch.  When a change is detected the following process is triggered:


Objective: Validate that upstream trunk still builds OK with current packaging for Ubuntu.

  1. A new snapshot upstream tarball is generated based on the latests commit to the upstream component.
  2. The latest archive packaging for the component is pulled in from lp:~ubuntu-server-dev/<COMPONENT>/essex
  3. Any changes in the testing packaging for the component are merged from lp:~openstack-ubuntu-testing/<COMPONENT>/essex
  4. New changelog entries are automatically created for the new upstream commits.
  5. The source package is generated and built in a clean schroot using sbuild locally.

On the assumption that the package built OK locally:

  1. The source package is uploaded to the Testing PPA (ppa:openstack-ubuntu-testing/testing)
  2. The testing packaging branch is push back to lp:~openstack-ubuntu-testing/<COMPONENT>/essex.
  3. The binary packages from the sbuild are installed into the local reprepro managed archive.

This process is managed by a single script (tarball.sh); Credit to Chuck Short for pulling together this part of the process based on work from Openstack upstream.

For changes to the nova project the deploy phase is then executed.


Objective: Validate that packages install, can be configured and reach a know good state prior to execution of testing.

This phase of testing uses Juju with Cobbler to deploy Openstack into the QA lab infrastructure; It utilizes branches of the Openstack charms to support use of a local archive along with a deployer wrapper around Juju written by Adam Gandelman which executes the actual deployment using Juju and monitors for errors.

The deployer is configured to know where to get the right codebase for the Openstack charms, which services to deploy and which relations to setup between services. As you can see from the above diagram this is non-trivial but the charms and Juju do most of the hard work.

Once Openstack is deployed successfully the test phase is then executed.


Objective: Validate that the Openstack deployment in the lab actually works!

At this point, we can run any integration tests we wish against the newly deployed cloud.  This testing is able to help us achieve multiple goals:

  • Early detection of upstream bugs that break Openstack functionality on Ubuntu
  • Verification that packaging branches in the development version of Ubuntu are compatible with upstream trunk.
  • Using these packages, verification that our Juju charms are deploying a functional Openstack cloud and are up-to-date with any deployment-related configuration changes upstream.

At the moment this phase looks like this:

  1. Configure the Openstack deployment (Adams deployer script provides some utility functions for locating specific services in the environment)
    • Creates network configuration in Nova for the private instance network as well as a pool of public floating IPs.
    • Upload an image into the Glance server for use during testing
    • Creates EC2 credentials in the Keystone server for use during testing.
  2. Run the devstack exercise test scripts which ensure basic functionality of the deployment. Currently, this includes:
    • Basic euca-tools EC2 API for starting and stopping instances
    • EC2 AMI bundle uploads
    • Floating IP allocation, association and connectivity to instance
    • Volume creation and attachment to instance

Note: These are the same sets of tests that are currently run against proposed commits to gerrit upstream.

Longer term we aim to use the Openstack Tempest test suite in the lab; Adam is currently working on getting this up and running.


The Jenkins instance in the QA lab is not publicly accessible; however all jobs run in the lab are published out (using the Jenkins build-publisher plugin) to http://jenkins.qa.ubuntu.com so that people can see the current state of the testing packaging in Ubuntu precise.

We are also working on setting up email notifications.

Success so far

Juju charms deploy Openstack components in a configuration that is compatible with upstream trunk prior to updates to packaging in Ubuntu.  Previously packages were updated in the archive first while Juju charm updates lagged behind as incompatibilities were uncovered after the fact.

We enabled automated testing 2 days prior to the 3rd Essex milestone release.  We were able to uncover and help fix a handful of bugs upstream before the release, including critical bugs like 921784.  In the past, these bugs were typical uncovered after the release (both upstream and in Ubuntu).

Since E3, there have been even more critical bugs uncovered by this testing and fixed upstream, some of which are only applicable to Ubuntu-specific configurations (not tested upstream) and would have been uncovered by users after code hit the Ubuntu archive (See 922232).

Further Plans for the Lab

Pre-commit  testing of changes to stable branches;  The Ubuntu Server team are  working upstream on maintaining the stable branches of released versions  of OpenStack – this work will validate patches proposed to stable  branches in review.openstack.org against the current version of the  packaging in released versions of Ubuntu.  Initially this will target  Diablo on Ubuntu 11.10 but will also support Essex on Ubuntu 12.04 once  released.  Ideally the testing process will provide feedback on  review.openstack.org to help the stable release team review proposed  patches.


Jenkins job configurations: lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Scripts supporting the lab: lp:~openstack-ubuntu-testing/+junk/jenkins-scripts

LVM snapshot preseeds and Cobbler snippets: lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot

All other relevant scripts, charm branches, etc: https://code.launchpad.net/~openstack-ubuntu-testing/


Overall management of delivery and general whip cracking: Dave Walker

Lab installation and base configuration: Pete Graner, Tim Gardner, Brad Figg, James Page

Fence agent for network power control of servers: Andres Rodriguez

Source package creation and build process: Chuck Short and James Page

Deployment testing using Juju: Adam Gandelman

Testing of Openstack: Adam Gandelman

Jenkins packaging, configuration and management: James Page

Gerrit Plugin for pre-commit testing and generally great ideas: Monty Taylor and James Blair

Writing and reviewing this post: Adam Gandelman, Chuck Short and Dave Walker.

Comment » | Uncategorized

OpenStack Jenkins dashboard available for testing Ubuntu snapshots

January 27th, 2012 — 7:50am

The keener eyed of you may have noticed:


James Page has setup the jobs in the Ubuntu OpenStack QA Lab to start publishing to the public Jenkins QA instance this morning. We now have automated build testing of all core OpenStack components triggered from upstream trunk commits. This is followed by automated deployment (-deploy) of OpenStack in the lab with a serving of testing (-test) once its all up and running.

Credit to Adam Gandelman for the Juju charm work, deployment framework and test execution and to Chuck Short for the hugely misnamed tarball.sh script which completes the git/bzr/packaging fu to build and deploy OpenStack packages!

The plan is to get the upstream Tempest test suite running in the lab; at the moment we are running a more limited test script just to ensure that you can spin up and instance and see it on the network.

(Crossposted from cloud.ubuntu.com)

1 comment » | Development

Mirantis: OpenStack Deployment on Fedora using Kickstart

May 18th, 2011 — 2:32pm

The team at Mirantis published a new blog post today on deploying OpenStack on Fedora (http://mirantis.blogspot.com/2011/05/openstack-deployment-on-fedora-using.html). From the blog:

In this article, we discuss our approach to performing an Openstack installation on Fedora using our RPM repository and Kickstart. When we first started working with OpenStack, we found that the most popular platform for deploying OpenStack was Ubuntu, which seemed like a viable option for us, as there are packages for it available, as well as plenty of documentation. However, our internal infrastructure is running on Fedora, so instead of migrating the full infrastructure to Ubuntu, we decided to make OpenStack Fedora-friendly. The challenge in using Fedora, however, is that there aren’t any packages, nor is there much documentation available. Details of how we worked around these limitations are discussed below.

The complete blog post with detailed step by step directions at http://mirantis.blogspot.com/2011/05/openstack-deployment-on-fedora-using.html.

Comment » | community, Development, Partner

Community Weekly Newsletter (May 7 – 13)

May 13th, 2011 — 11:01am

OpenStack Community Newsletter – May 13, 2011

This weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please email [email protected].

Developer Lounge at OpenStack Conference and Design Summit






  • Data Tracking Graphs – http://wiki.openstack.org/WeeklyNewsletter
  • OpenStack Compute (NOVA) Data
    • 41 Active Reviews
    • 268 Active Branches – owned by 72 people & 14 teams
    • 952 commits by 74 people in last month
  • OpenStack Object Storage (SWIFT) Data
    • 6 Active Reviews
    • 63 Active Branches – owned by 21 people & 6 teams
    • 164 commits by 16 people in last month
  • OpenStack Image Registry (GLANCE) Data
    • 3 Active Reviews
    • 22 Active Branches – owned by 7 people & 4 teams
    • 89 commits by 10 people in last month
  • Twitter Stats for Week:  #openstack 244 total tweets; OpenStack 1,122 total tweets  (does not include RT)
  • Bugs Stats for Week: 336 Tracked Bugs; 95 New Bugs; 44 In-process Bugs; 1 Critical Bugs; 17 High Importance Bugs; 179 Bugs (Fix Committed)
  • Blueprints Stats for Week:  215 Blueprints; 15 Essential, 21 High, 14 Medium, 18 Low, 147 Undefined
  • OpenStack Website Stats for Week:  15,167 Visits, 34,430 Pageviews, 58.69% New Visits
    • Top 5 Pages: Home 41.94%; /projects 10.19%; /projects/compute 15.98%; /projects/storage 11.30%; /community 5.62%


Comment » | Communication, Newsletter

Ubuntu Cloud Days IRC Event

March 21st, 2011 — 2:29pm

On March 23 and 24th, the Ubuntu community is hosting their first Cloud Days event via IRC. Soren Hansen is representating the OpenStack Community with an introduction talk at 20.00 UTC. The schedule for Wednesday March 23 is below:

Wed 23rd Mar
16.00 UTC
Cloud Computing 101, Ask your questions — kim0
17.00 UTC Scaling shared-storage web apps in the cloud with Ubuntu & GlusterFS — semiosis
18.00 UTC What is Ensemble? – Presentation and Demo — SpamapS
19.00 UTC Using Linux Containers in Natty — hallyn
20.00 UTC Open-Stack Introduction — soren

For all the details, please visit http://cloud.ubuntu.com/2011/03/ubuntu-cloud-days-approaching/.

1 comment » | Communication, Event, Partner

OpenStack Weekly Newsletter (January 29 – February 4)

February 4th, 2011 — 10:19am

OpenStack Community Newsletter – February 4, 2011

This weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please email [email protected].






  • Data Tracking Graphs – http://wiki.openstack.org/WeeklyNewsletter
  • OpenStack Compute (NOVA) Data
    • 6 Active Reviews
    • 132 Active Branches – owned by 40 people & 9 teams
    • 1,001 commits by 60 people in last month
  • OpenStack Object Storage (SWIFT) Data
    • 5 Active Reviews
    • 44 Active Branches – owned by 19 people & 3 teams
    • 285 commits by 14 people in last month
  • Twitter Stats for Week:  #openstack 91 tweets; 87 re-tweets; all OpenStack total tweets 1008
  • Bugs Stats for Week:  365 Tracked Bugs; 57 New Bugs; 16 In-process Bugs; 17 Critical Bugs; 48 High Importance Bugs; 247 Bugs (Fix Committed)
  • Blueprints Stats for Week:  158 Blueprints; 10 Essential, 6 High, 12 Medium, 16 Low, 114 Undefined
  • OpenStack Website Stats for Week:  7,881 Visits, 18,389 Pageviews, 62.33% New Visits
    • Top 5 Pages: Home 40.46%; /projects 11.69%; /projects/compute 19.37%; /projects/storage 13.54%; /Community 6.35%


Comment » | Communication, community, Newsletter

OpenStack Weekly Newsletter (January 7 – 14)

January 14th, 2011 — 9:17am

OpenStack Community Newsletter – January 14, 2011

This weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please email [email protected].






  • Data Tracking Graphs – http://wiki.openstack.org/WeeklyNewsletter
  • OpenStack Compute (NOVA) Data
    • 10 Active Reviews
    • 101 Active Branches – owned by  32 people &  6 teams
    • 1,118 commits by 60 people in last month
  • OpenStack Object Storage (SWIFT) Data
    • 2 Active Reviews
    • 36 Active Branches – owned by  19 people & 2 teams
    • 181 commits by  12 people in last month
  • Twitter Stats for Week:  #openstack 77 tweets; 48 re-tweets; all OpenStack total tweets 542
  • Bugs Stats for Week:  225 Tracked Bugs; 50 New Bugs; 15 In-process Bugs; 7 Critical Bugs; 27 High Importance Bugs; 85 Bugs (Fix Committed)
  • Blueprints Stats for Week: 125 Blueprints; 1 Essential, 6 High, 14 Medium, 13 Low, 91 Undefined
  • OpenStack Website Stats for Week:  11,076 Visits, 24,838 Pageviews, 70.31% New Visits
    • Top 5 Pages: Home 43.27%; /projects 12.02%; /projects/compute 18.29%; /projects/storage 12.71; /Community 5,40%


1 comment » | Communication, Newsletter

Back to top