The OpenStack Blog

Category: Tools

Infrastructure Bootcamp

Recently the Project Infrastructure team hosted a two-day “bootcamp”
for people who are interested in contributing to the Project
Infrastructure.  The OpenStack project is so large, and continuing to
grow, that creating and operating the developer infrastructure for the
project itself is a unique challenge.  Because OpenStack receives code
contributions from more than 600 developers in a release cycle, and
merges as many as 200 changes per day, we are at the cutting edge of
distributed development and testing.

The project infrastructure covers a wide range of tools and services
used by the project, including code review, testing, and
collaboration.  The design and operation of these systems is managed
under the Infrastructure Program which is overseen by the TC just like
the rest of the OpenStack project.  And like any other OpenStack
program, our team is open and we welcome contributions from anyone.

Managing the infrastructure for a project of this scale is a lot of
work, but it is uniquely rewarding because it affects every OpenStack
project and affords interactions with all of the developers.  We heard
a lot of interest from persons and companies who wanted to contribute,
so we held the bootcamp to get anyone who was interested in
contributing together in a room with the current core team of
infrastructure developers.

OpenStack Infrastructure Bootcamp kicks off

OpenStack Infrastructure Bootcamp kicks off
Photo: Elizabeth Krumbach, CC-BY 2.0

Day one saw us all together and discussing how contributions are
accepted, how the team operates, and how new members can expect to
become core members in the future.  We discussed each major system at
a high level, and worked out how all of the systems interact with each

In the evening, we all got together for dinner and spent several
enjoyable hours talking about how we could improve the system, and
generally getting to know each other.

On day two, we dove deeper into some topics of particular interest to
attendees, and generally had a less structured approach where people
who shared interest in an area got together and discussed it in depth.

I think everyone who attended got a lot out of the event, and we’re
already seeing significant new contributions as a result.  I hope that
in several months more time we will have new core members on our
team.  I also think this is a good model for other programs in
OpenStack who want to quickly bring new contributors up to speed.

I’d like to thank Monty Taylor for organizing the event, the rest of
the core contributors (or “coremudgeons”) for talking about what we do
for two days straight, and Hewlett-Packard and the OpenStack
Foundation for sponsoring the event.

Some reactions from others who attended:

Introducing the OpenStack Activity Board

I am pleased to announce that a beta release of the OpenStack Activity Board (beta) is now live. The development Activity Board announced few months ago provides a visual overview of all the OpenStack public activity of community members across multiple dimensions: contributors and organizations, projects and tools. From a single interface, you can easily surf OpenStack project content, whether it is coming from the LaunchPad bug tracker, Git or Gerrit, all mapped against the OpenStack Foundation members database.

The Vision Behind the Activity Board

The intention is to give the community a way to answer very precise questions like: who’s contributing to that particular feature of OpenStack? What is that developer/company working on? Which commits/changes are related to a particular bug? Who’s joined the development community recently? With the Activity Board, we have integrated information across the different systems used to develop OpenStack to give corporate and community users a unified view of all the efforts going into OpenStack. There are two main parts of the Activity Board: the Dash and the Insights. The Dash contains reports built by Bitergia using the free software suite MetricsGrimoire, it focuses on trends and quantitative presentation. The Insights, enriched with a semantic layer, powered by zAgile’s open source Wikidsmart adds qualitative details with faceted search of concepts across all the different repositories, tracing people and artifacts across different repositories and bug tracker in order to reconcile people and corresponding contributions.

Looking to the Future

With the integration, we know that we can become a much more efficient project in terms of communicating to members, monitoring our progress, and getting work done. The project is its infancy and we wish to continue to evolve the solution and enhance it with everyone’s feedback. First of all we need you to look at the data and let us know if you find any mistake (we’re keeping a log of known issues). There are a number of areas we want to consider, for example:

  • Streamline the faceted search interface according to the community’s desires
  • Add more reports based on community requests
  • Add integration with other OpenStack project data sources (blueprints, mailing lists, Ask, user groups, IRC)
  • Potentially enable interoperability (which is inherent in the Wikidsmart abilities) so that an event in one tool, for example LaunchPad, kicks off another action or update within another tool
  • Remove the dependency on Confluence, port the visualization to an open source wiki or portal,
  • Publish all the code and tools, integrate the Activity board in the OpenStack infrastructure processes

Learn More

If you would like to learn more, we are featuring the OpenStack Activity Board in the upcoming events:

I look forward to your comments and further collaboration to make sure this solution benefits the whole community to the maximum extent.

Hacking on Ebooks

Gentlemen prefer PDF, according to Tim O’Reilly’s data from Rough Cuts five years ago. At OpenStack we see some preference for PDF, though there are three times as many visits to the HTML version of our Compute Admin manual. Still, the PDF version of the guide is downloaded about five times a day. I do believe that gentlemen prefer PDF or some sort of book-like reading material. When asked, readers cite portability and search scope as two benefits to the form. However, as David Cramer, our doc tools developer put it at our recent hackathon, “PDFs are like cement.” With the boom of mobile and tablet screens, a stretchy and flexible screen-reader format like epub fills a need – we need content that works well on the 200 plus devices that fit into one hand.

So on 11/11/11, in the Austin Rackspace office, we did some hacking to be able to create epub output from our DocBook source files. I blogged about it for the OpenStack Planet blog from my blog, DocBook, ePub, Hackathon, what more could you ask for? prior to the event, talking about some of our prep work.

I’m pleased to show you the results – we did get output for epub and also tested the Mobi output on a Kindle, all in one day, with a team of about seven hackers including developers, writers, and testers.

We first tested the process using built-in epub transforms that ship with Oxygen, our XML editor, who supports open source projects like OpenStack by donating licenses to documentation contributors. Thank you Oxygen! We were able to use that output to start testing. Here’s our white board with the list of bugs.

While the writers and testers were hacking on output, programmers were working on ensuring we could get the epub output through Apache Maven, our build tool. By the end of the day, we could output epub through our automatic build process also!

As happens with hackathons, there’s some cleanup work to do – for example, our neat-o dynamic SVG cover page that takes in variables like the book title doesn’t output a cover for the epub. Also, most “real” epub output workflows convert tables from text to image (I know, crazy huh, when you think about the loss of search capability), but the tables in epub output act a little funky when resizing. Also, mobi, the Kindle format, has a problem with the way lists are marked up, but these are fixable and on the bug log.

I haven’t decided yet whether the epub output is high quality enough to offer it for download for every book on the OpenStack docs site, nor do I know if there’s much demand for the output, but I’d like to offer the OpenStack Starter Guide as an epub download. The team at CSS OSS works hard on this content, and I’d like to see it get spread onto many devices. Let me know how well it works for you and if you think epub has a place as a regular output for OpenStack documentation.

Have you visited the lately?

A few weeks ago we posted the results of the OpenStack Social Media survey. Thirty percent of survey  respondents indicated that they keep up on OpenStack developments and news on I thought now would be a good time to remind everyone of what a valuable addition the forum is to the OpenStack community, and it has only been up and running since May!

Currently the NOVA board is the most heavily used project board, and admin Jordan Rinke (@JordanRinke) has been especially active in answering  questions regarding installation. Stephen Spector has also been experimenting with using the Events category on the General board to help plan upcoming OpenStack Events like this October’s conference in Boston.  If you have suggestions please feel free to chime in on the forum.

There are currently 85 registered users and I encourage anyone with a question, comment, or insight to complete the easy sign up. The more members and participation the forum sees the more useful it will be!

OpenStack Social Media Survey Results

Over the past week, the OpenStack Community Management team has conducted an online Social Media Engagement survey to better understand the needs and wants of various community member types in relation to the information available. The survey results are available, SurveySummary_06082011 with no details on the participants who answered questions.

Based on these results, we are taking the following three actions to meet the needs of the community:

1. Result – and the OpenStack presence on various social networks (especially Twitter and LinkedIn) are the primary information sources of most respondents with 60% looking to receive even more information on a regular basis.

    Action –  A social network plan will be drafted to better link the various OpenStack information repositories with community member information access points. This plan will drive content from OpenStack Slideshare, Vimeo, Flickr, and other repositories to Twitter, LinkedIn, Facebook, etc. Be on the lookout for this plan as we intend to ask the broader community for feedback on the various options available in this plan.

    2. Result – Two community member types stood out in the survey: Participating company prospects and Active contributor prospects.

    Action – A “Getting Started with OpenStack” document is planned for publication with all the information on becoming a participating company, active participant, developer, etc. This document will be visible in many social networking sites including the OpenStack home page to provide critical information for these community prospects.

    3. Result – The OpenStack Forum is showing strength for people looking for information and answers to common OpenStack user questions.

    Action – New marketing and promotion initiatives will be started to significantly raise the awareness of the OpenStack Forum and help drive more participation.

    Thanks again for everyone that participated and if you have more thoughts on this issue, please contact Summer Fouche; or Stephen Spector.

    10 Steps to Initiating an OpenStack Cloud Service

    This blog post originates from Eric Day at I have copied it here for your reference:

    OpenStack currently consists of three main components: Nova (Compute), Swift (Object Storage), and Glance (Image Service). There are some other projects such as a dashboard and mobile apps as well. You can see the full list here. This is great start, but in order for OpenStack to compete long term other infrastructure and platform services will need to be brought in. I’d like to talk about the process I’m taking with a new message queue service.

    Step 1 – Idea

    The first step is to figure out what is missing. What new service would compliment the software already available? What hasn’t been solved yet? What are users asking for? A message queue seemed like an appropriate next step as most applications that need to scale out and be highly available will make use of a queue at some point (sometimes not in the most obvious form). It will also allow other cloud services to be built on top of it. In fact, the current OpenStack projects could even leverage a queue service for new features.

    Step 2 – Initial requirements

    Before you write up a proposal and send it out, it might be a good idea to gather some initial requirements and figure out what it may look like. Don’t worry about details as the community will help flush this out later. Some of the major requirements when thinking about OpenStack projects are horizontal scalability, multi-tenancy, modular API, REST API, zones and locality awareness, and no single points of failure (high availability). This is a pretty heavy set of requirements before even getting into service specifics, but this will help you think about how to approach a service. You may have to diverge away from traditional thinking for a particular service. For example, what worked in a rack or a data center may not work in the cloud. You need to account for this up front and state behavioral differences from what folks may expect. For the queue service, this meant not taking a traditional approach you see in some queue protocols and services, and instead integrating ideas from distributed services.

    A multi-tenant cloud is a very different environment from what many people are used to and usually requires a different approach to solve problems. If folks tell you you’re re-inventing the wheel, take their concerns into consideration, but also realize you may not be. You may be writing a jet engine.

    Step 3 – Wiki and Mailing List Proposal

    Once you have a good idea and a rough outline, you’ll probably want to run it by a couple people for feedback before sending it to everyone. You’ll then want to create a new wiki page on the OpenStack wiki and send a note to the public mailing list that mentions the wiki page and asks for community feedback. For example, the queue service proposal I wrote can be found here. There is an enormous amount of collective experience and brain power on the mailing list which will help point out any issues with the proposal. The service you initially propose may look nothing like the service you actually build. It’s also quite possible the service you propose is not a good fit for the cloud or OpenStack. The community will help iron all these details out.

    Step 4 – Wait

    It can take folks a while to catch up on public mailing lists, so be patient. Let people know about the proposal by other means (blog, tweet, irc, …) and help facilitate the conversation as people respond.

    Step 5 – Prototype

    Once you feel the community is content with the proposal and it’s a viable idea (don’t expect consensus), prototype it! This shows the community you are serious and this exercise will help work out more issues in the proposal. Let the community know about it and again wait for any feedback. This doesn’t need to be anything fancy, for the queue service I put this together over a weekend.

    Step 6 – Name and Language

    Now comes the difficult part, choosing a project name. I’d suggest not using the mailing list for this as it will be a lot of noise for a matter that isn’t too important. Ask a couple folks who may also be interested for ideas and make sure it’s not already taken (search on github, Launchpad, Google, etc). For the queue service we decided on “Burrow”.

    You’ll also need to figure out the most appropriate language. For middleware and services, Python is a good default. If efficiency is a concern, look at Erlang or C/C++. Be sure to send another mail to the list and ask for feedback. With the queue service I initially proposed C++ with Erlang as an alternative since efficiency is a major concern (especially around utilizing multiple cores), and the community came back mixed but with more enthusiasm for Erlang.

    Step 7 – Bootstrap the Project on Launchpad

    We’re using Launchpad for OpenStack project management. You’ll need to create a project and a number of groups to manage it. For example, the queue project can be found here. The groups have the following roles (replace burrow with your project name):

    • burrow – Public group that anyone can join. This currently includes members on the main OpenStack mailing list, but we’re setup this way in case we need to break projects out into their own list.
    • burrow-drivers – The group responsible for maintaining the project, managing blueprints, and making releases.
    • burrow-core – The group responsible for performing code reviews.
    • burrow-bugs – The group responsible for managing bugs.

    Step 8 – Lock onto Releases and Milestone Schedule

    While not important right away, it might be a good idea to start working with the OpenStack release cycle. Releases are currently every three months with milestones setup in each release for feature freeze, bug freeze, and releases. See the release page for more details. Launchpad makes it fairly simple to manage this, you’ll just want to create a new series (for example, “cactus” right now), and a couple milestones within that series for the freezing and release. Ask on the mailing list or on IRC if you need any help, but a good rule of thumb is to follow what other established projects do (like Nova).

    Step 9 – Code!

    Get to work and try to recruit other developers to help you. Keep the community updated with progress by using IRC, mailing list,, and tweets.

    Step 10 – Submit to the Project Oversight Committee

    Up until this point your project has not been an official OpenStack project. It is a well thought-out idea driven by the community that probably has a good chance though. You’ll need to make a proposal to the POC using this page once the project can stand on it’s own. You probably don’t need a final version, but you need something that is functional and more robust than a prototype. The POC meets weekly, although it may take more time (and some conversations) to decide if your project is ready. The queue service I’ve been driving has not been proposed since it’s not ready, so you may want to take all this with a grain of salt. It is my hope to have the first version ready to propose in April as part of the Cactus release.

    Final Thoughts

    This process will vary and can certainly be refined. I’m stating what I’ve done with a new project, but existing projects will obviously need to take a different route. The main idea to keep in mind though is that any OpenStack project should be seen as community driven, not just by an individual or company. One or more individuals may carry out a large part of the work of the community initially, but community concerns and feedback should always be taken with the utmost importance.

    Annoucing Dashboard for OpenStack

    Devin Carlen posted information to the OpenStack community on the release of a new Dashboard for OpenStack at The complete posting is also reproduced here:

    I’d like to take this opportunity to formally announce the Dashboard for OpenStack. It’s been available on Launchpad for a little while now so I could gather a bit of initial feedback about it. Enough people have played with it and found it useful that it makes sense to unveil it to a larger audience.

    The  Dashboard for OpenStack is based primarily on code that was developed for the NASA Nebula Dashboard. We received the green light to release the code under the Apache license and have done so. The project is based on Django and Python and consists of two primary pieces:


    This is a Django module that contains all of the interesting bits. It’s designed to be reusable and modular so it can be used in a variety of projects. The NASA Nebula Dashboard uses this module, as does the OpenStack Dashboard.

    The repo is available at:

    As of Ubuntu Natty, it will be available in the apt repo:


    This is a Django site that provides a bare minimum reference implementation around django-nova. This is essentially just CSS, django-registration for creating accounts, and the settings required to make django-nova function properly. The goal of this site is to provide something demoable with some OpenStack branding on it. Other organizations wanting to deploy a dashboard will want to roll their own, but using this as a reference.

    The repo is available at:


    Migrate to OpenStack API

    Currently django-nova is based on the EC2 API for several reasons:

    * OpenStack API lacks support for Volumes (this is being remedied very soon)
    * OpenStack API has some conflicts with how project and user groups are handled.
    * OpenStack API lacks admin API functions such as creating users and projects, starting VPNs.
    We have cobbled together the admin functions in the EC2 based admin endpoint for now.

    The goal is to transition to the OpenStack API as soon as it is possible. There’s no harm in starting this effort in a separate branch now so we can begin figuring out exactly where the pain points will be.

    Combine django-nova and openstack-dashboard repos

    Since these are complementary projects, I originally created them as separate Launchpad repos, but this created a lot of extra overhead when applying fixes that related to both projects. The code won’t be combined anymore than it already is, but I will be restructuring the openstack-dashboard repo to include django-nova in a separate folder. This will make it much easier to deal with.

    Improve user experience

    We followed an agile development process and created the minimum of what was needed as we proceeded, but for the most part the functionality has stabilized. There is room for improvement in the general usability of the sight, such as adding more client side scripting, improving layouts, etc.

    There is much more to be done, but this is a good starting point for discussion!

    Chef Meets the iPad with OpenStack

    Since OpenStack was announced, we’ve seen a flurry of activity in the developer community. Hundreds of people have made over a thousand contributions and we couldn’t be more excited.

    Today I’d like to share one of the more unconventional contributions with you. Most of the development activity in OpenStack is in the storage and compute systems, but we also have front end projects for people who use OpenStack and need an easy way to manage their cloud resources.

    One of these projects is the OpenStack iPad app, which is based on the Rackspace Cloud Pro iPad app that is currently in the App Store. This app uses the OpenStack compute and storage APIs to help you manage your cloud resources, and offers a few features outside of the scope of the APIs, such as viewing RSS system status feeds, pinging your compute nodes from several locations around the world, and emailing files from OpenStack Object Storage.

    Our newest feature, made possible by working with people from Opscode, is integration with Chef and the Opscode Platform. Before, when you created a compute node on the iPad, you would then need to log into the node and manually install any software, data, and configurations you may need. This could take a considerable amount of time and effort for some deployments, but not anymore. If you’re a Chef user, you can configure the iPad app with your Chef or Opscode credentials to automatically run any recipes or role assignments immediately after the node is provisioned. Look ma, no shell!

    Let’s suppose you’re running a Cassandra cluster to horizontally scale your data, and you’re out of town on a business trip. Your laptop is in your hotel room, and you’re stuck in a conference room across town with only your iPad. Then, your monitoring service sends you an SMS telling you that all of your Cassandra machines are running at capacity. But you were smart; you used Chef to automate your infrastructure by creating roles for all of your compute nodes. You simply log into the iPad app and provision a new node with “role[cassandra]“ as your run list. The machine boots up and adds itself to your Cassandra cluster without requiring anything else from you. Awesome!

    I’ve included a video below showing you the entire process as I bootstrap a compute node running Apache 2 and MySQL. I hope you enjoy it, and if you have any questions or want to talk about contributing to the OpenStack iOS apps, get in touch with me via email at [email protected], Twitter at @greenisus, or on the phone at 901-299-9306. Even if you’re new to Objective-C and Cocoa, I’ll be happy to help you get started and learn everything you need to know to help make the OpenStack apps even better.

    Back to top