10 Steps to Initiating an OpenStack Cloud Service

This blog post originates from Eric Day at http://oddments.org/?p=591. I have copied it here for your reference:

OpenStack currently consists of three main components: Nova (Compute), Swift (Object Storage), and Glance (Image Service). There are some other projects such as a dashboard and mobile apps as well. You can see the full list here. This is great start, but in order for OpenStack to compete long term other infrastructure and platform services will need to be brought in. I’d like to talk about the process I’m taking with a new message queue service.

Step 1 – Idea

The first step is to figure out what is missing. What new service would compliment the software already available? What hasn’t been solved yet? What are users asking for? A message queue seemed like an appropriate next step as most applications that need to scale out and be highly available will make use of a queue at some point (sometimes not in the most obvious form). It will also allow other cloud services to be built on top of it. In fact, the current OpenStack projects could even leverage a queue service for new features.

Step 2 – Initial requirements

Before you write up a proposal and send it out, it might be a good idea to gather some initial requirements and figure out what it may look like. Don’t worry about details as the community will help flush this out later. Some of the major requirements when thinking about OpenStack projects are horizontal scalability, multi-tenancy, modular API, REST API, zones and locality awareness, and no single points of failure (high availability). This is a pretty heavy set of requirements before even getting into service specifics, but this will help you think about how to approach a service. You may have to diverge away from traditional thinking for a particular service. For example, what worked in a rack or a data center may not work in the cloud. You need to account for this up front and state behavioral differences from what folks may expect. For the queue service, this meant not taking a traditional approach you see in some queue protocols and services, and instead integrating ideas from distributed services.

A multi-tenant cloud is a very different environment from what many people are used to and usually requires a different approach to solve problems. If folks tell you you’re re-inventing the wheel, take their concerns into consideration, but also realize you may not be. You may be writing a jet engine.

Step 3 – Wiki and Mailing List Proposal

Once you have a good idea and a rough outline, you’ll probably want to run it by a couple people for feedback before sending it to everyone. You’ll then want to create a new wiki page on the OpenStack wiki and send a note to the public mailing list that mentions the wiki page and asks for community feedback. For example, the queue service proposal I wrote can be found here. There is an enormous amount of collective experience and brain power on the mailing list which will help point out any issues with the proposal. The service you initially propose may look nothing like the service you actually build. It’s also quite possible the service you propose is not a good fit for the cloud or OpenStack. The community will help iron all these details out.

Step 4 – Wait

It can take folks a while to catch up on public mailing lists, so be patient. Let people know about the proposal by other means (blog, tweet, irc, …) and help facilitate the conversation as people respond.

Step 5 – Prototype

Once you feel the community is content with the proposal and it’s a viable idea (don’t expect consensus), prototype it! This shows the community you are serious and this exercise will help work out more issues in the proposal. Let the community know about it and again wait for any feedback. This doesn’t need to be anything fancy, for the queue service I put this together over a weekend.

Step 6 – Name and Language

Now comes the difficult part, choosing a project name. I’d suggest not using the mailing list for this as it will be a lot of noise for a matter that isn’t too important. Ask a couple folks who may also be interested for ideas and make sure it’s not already taken (search on github, Launchpad, Google, etc). For the queue service we decided on “Burrow”.

You’ll also need to figure out the most appropriate language. For middleware and services, Python is a good default. If efficiency is a concern, look at Erlang or C/C++. Be sure to send another mail to the list and ask for feedback. With the queue service I initially proposed C++ with Erlang as an alternative since efficiency is a major concern (especially around utilizing multiple cores), and the community came back mixed but with more enthusiasm for Erlang.

Step 7 – Bootstrap the Project on Launchpad

We’re using Launchpad for OpenStack project management. You’ll need to create a project and a number of groups to manage it. For example, the queue project can be found here. The groups have the following roles (replace burrow with your project name):

  • burrow – Public group that anyone can join. This currently includes members on the main OpenStack mailing list, but we’re setup this way in case we need to break projects out into their own list.
  • burrow-drivers – The group responsible for maintaining the project, managing blueprints, and making releases.
  • burrow-core – The group responsible for performing code reviews.
  • burrow-bugs – The group responsible for managing bugs.

Step 8 – Lock onto Releases and Milestone Schedule

While not important right away, it might be a good idea to start working with the OpenStack release cycle. Releases are currently every three months with milestones setup in each release for feature freeze, bug freeze, and releases. See the release page for more details. Launchpad makes it fairly simple to manage this, you’ll just want to create a new series (for example, “cactus” right now), and a couple milestones within that series for the freezing and release. Ask on the mailing list or on IRC if you need any help, but a good rule of thumb is to follow what other established projects do (like Nova).

Step 9 – Code!

Get to work and try to recruit other developers to help you. Keep the community updated with progress by using IRC, mailing list, planet.openstack.org, and tweets.

Step 10 – Submit to the Project Oversight Committee

Up until this point your project has not been an official OpenStack project. It is a well thought-out idea driven by the community that probably has a good chance though. You’ll need to make a proposal to the POC using this page once the project can stand on it’s own. You probably don’t need a final version, but you need something that is functional and more robust than a prototype. The POC meets weekly, although it may take more time (and some conversations) to decide if your project is ready. The queue service I’ve been driving has not been proposed since it’s not ready, so you may want to take all this with a grain of salt. It is my hope to have the first version ready to propose in April as part of the Cactus release.

Final Thoughts

This process will vary and can certainly be refined. I’m stating what I’ve done with a new project, but existing projects will obviously need to take a different route. The main idea to keep in mind though is that any OpenStack project should be seen as community driven, not just by an individual or company. One or more individuals may carry out a large part of the work of the community initially, but community concerns and feedback should always be taken with the utmost importance.

Tags:

OpenStack Conference / Design Summit Update

Here is the latest information on the upcoming OpenStack Conference / Design Summit:

Dates: April 26-29, 2011
Location: Hyatt Regency in Santa Clara, CA (http://santaclara.hyatt.com/hyatt/hotels/index.jsp?null)
Cost: $100 for all attendees; late registrations will pay $150
Event Size: 350 attendees is the maximum

The OpenStack Conference will be April 26-27, 2011 for business development, technologists and others interested in the OpenStack open source project. The OpenStack Design Summit for community developers will be April 27-29, 2011 and will consist of three days of blueprint reviews and discussions for future OpenStack releases. The agenda for both events is still in process with a list of currently submitted topics at http://etherpad.openstack.org/SubmittedTopics and working agenda at http://etherpad.openstack.org/DesignSummitApril2011. The Program Committee is currently looking at the submitted topics and fitting them into the agenda as well as recruiting more speakers/topics.

The event registration website is currently being completed and is expected to be published by the middle of the week. The registration site will also contain a link to the Hyatt Regency so attendees can take advantage of the discounted room block. If you are staying at the Hyatt, please leverage the room block as the community is required to cover the costs of all rooms not booked which could be a significant budget item.

Finally, several community participating companies have stepped up to sponsor the event and that list will be released soon. If you are interested in sponsoring, please review the current prospectus at http://www.box.net/shared/ng0r7u6n44.

If you have additional questions, want to speak at the event, or are interested in being a sponsor please contact me at [email protected].

Tags:

Annoucing Dashboard for OpenStack

Devin Carlen posted information to the OpenStack community on the release of a new Dashboard for OpenStack at https://lists.launchpad.net/openstack/msg01069.html. The complete posting is also reproduced here:

I’d like to take this opportunity to formally announce the Dashboard for OpenStack. It’s been available on Launchpad for a little while now so I could gather a bit of initial feedback about it. Enough people have played with it and found it useful that it makes sense to unveil it to a larger audience.

The  Dashboard for OpenStack is based primarily on code that was developed for the NASA Nebula Dashboard. We received the green light to release the code under the Apache license and have done so. The project is based on Django and Python and consists of two primary pieces:

django-nova

This is a Django module that contains all of the interesting bits. It’s designed to be reusable and modular so it can be used in a variety of projects. The NASA Nebula Dashboard uses this module, as does the OpenStack Dashboard.

The repo is available at: https://launchpad.net/django-nova

As of Ubuntu Natty, it will be available in the apt repo:

http://www.ubuntuupdates.org/packages/show/302171

openstack-dashboard

This is a Django site that provides a bare minimum reference implementation around django-nova. This is essentially just CSS, django-registration for creating accounts, and the settings required to make django-nova function properly. The goal of this site is to provide something demoable with some OpenStack branding on it. Other organizations wanting to deploy a dashboard will want to roll their own, but using this as a reference.

The repo is available at: https://launchpad.net/openstack-dashboard

——
Future
——

Migrate to OpenStack API

Currently django-nova is based on the EC2 API for several reasons:

* OpenStack API lacks support for Volumes (this is being remedied very soon)
* OpenStack API has some conflicts with how project and user groups are handled.
* OpenStack API lacks admin API functions such as creating users and projects, starting VPNs.
We have cobbled together the admin functions in the EC2 based admin endpoint for now.

The goal is to transition to the OpenStack API as soon as it is possible. There’s no harm in starting this effort in a separate branch now so we can begin figuring out exactly where the pain points will be.

Combine django-nova and openstack-dashboard repos

Since these are complementary projects, I originally created them as separate Launchpad repos, but this created a lot of extra overhead when applying fixes that related to both projects. The code won’t be combined anymore than it already is, but I will be restructuring the openstack-dashboard repo to include django-nova in a separate folder. This will make it much easier to deal with.

Improve user experience

We followed an agile development process and created the minimum of what was needed as we proceeded, but for the most part the functionality has stabilized. There is room for improvement in the general usability of the sight, such as adding more client side scripting, improving layouts, etc.

There is much more to be done, but this is a good starting point for discussion!

Tags:

OpenStack Developer Activity Review (February 21 – February 28)

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at [email protected]. I am always ready to listen to the community for new ideas.

Activities
• Nova 2011.1.1 release candidate available for testing – http://nova.openstack.org/tarballs/nova-2011.1.1~bzr653.tar.gz

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

Announcing the OpenStack Dashboard – Devin Carlen announced the availability of the OpenStack Dashboard based on the NASA Nebula Dashboard. Code links at https://lists.launchpad.net/openstack/msg01069.html.
Availability of RHEL build of Bexar release of OpenStack Nova – Grid Dynamics announced the availability of OpenStack Nova RHEL 6.0 build. The instructions to install on run on their build at http://wiki.openstack.org/NovaInstall/RHEL6Notes.
Working on fixing code after a review? Please mark merge proposal Work in Progress – Jay Pipes commented on the backlog of code waiting for review at https://code.launchpad.net/nova/+activereviews. Jay suggests several steps for developers whose code has been reviewed with suggested code fixes. Ewan Mellor suggested that a new Wiki page be created with all the information for developers on “How Peer Reviews Work”.
Should the OpenStack API re-use the EC2 credentials? – Justin Santa Barbara submitted a new blueprint, https://blueprints.launchpad.net/nova/+spec/authentication-consistency, to handle the authentication issues that arose between how OpenStack API and EC2 worked. More than 20 responses on this topic were posted so see https://lists.launchpad.net/openstack/msg00917.html for more information.
Decoupling of Network and Compute Services for the new Network Service design – Ryu Ishimoto proposes a possible solution to make NOVA no longer directly dependent on the network services to allow for future “pluggable” options. More than 20 responses on this topic as well – see https://lists.launchpad.net/openstack/msg00908.html for more information.

Statistics

  • Number of OpenStack Developers on Contributors List – 151
  • Cactus Release Status – Blueprints (http://wiki.openstack.org/releasestatus/)
    • Essential – 5 Design Approved; 1 Implemented – 1 Needs Code Review – 1 Beta Available
    • High – 11 Blueprints; 2 Drafting
    • Medium – 22 Blueprints; 3 Needs Code Review; 1 Blocked
    • Low – 16 Blueprints; 2 Needs Code Review; 4 Implemented

For the latest on development activities on OpenStack please check these sites for more details:

Tags:

Community Weekly Newsletter (February 18 – 25)

OpenStack Community Newsletter – February 25, 2011

This weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please email [email protected].

Coming to you 1 day early as I am flying back from Korea on Friday.

HIGHLIGHTS

EVENTS

DEVELOPER COMMUNITY

GENERAL COMMUNITY

COMMUNITY STATISTICS (2/18– 2/24)

  • Data Tracking Graphs – http://wiki.openstack.org/WeeklyNewsletter
  • OpenStack Compute (NOVA) Data
    • 29 Active Reviews
    • 167 Active Branches – owned by 46 people & 9 teams
    • 1,684 commits by 58 people in last month
  • OpenStack Object Storage (SWIFT) Data
    • 5Active Reviews
    • 53 Active Branches – owned by 19 people &   teams
    • 191 commits by 10 people in last month
  • Twitter Stats for Week:  #openstack  tweets;  re-tweets; all OpenStack total tweets  ** Looking for new tool to gather this data **
  • Bugs Stats for Week:  234 Tracked Bugs; 43 New Bugs; 32 In-process Bugs; 5 Critical Bugs; 23 High Importance Bugs; 74 Bugs (Fix Committed)
  • Blueprints Stats for Week:  165 Blueprints; 6 Essential, 15 High, 28 Medium, 24 Low, 92 Undefined
  • OpenStack Website Stats for Week: 7,111 Visits, 15,861 Pageviews, 52.90% New Visits
    • Top 5 Pages: Home 42.07%; /projects 11.73%; /projects/compute 17.23%; /projects/storage 12.12%; /Community 7.56%

OPENSTACK IN THE NEWS

Tags:

OpenStack in Korea

It’s been a very busy week for OpenStack in Koera and I wanted to highlight the events with some more details and pictures.

Monday Feb 21 – OpenStack at KT

Stephen Spector (OpenStack Community Manager) and John Willis from OpsCode presented to the Cloud Service Business Unit at KT on OpenStack, DevOps, and Chef. Unfortunately, the flag was cut off in the picture but OpenStack has made its mark on KT with a sizable banner hanging in their cloud computing offices.

Tuesday Feb 22 – OpenStack at Korea University

Professor Chuck Yoo invited KT and OpenStack for a special cloud computing presentation to the Graduate School Computer Science department. KT supported the OpenStack introduction presentation from Stephen Spector with detailed technical overviews of both OpenStack Compute and Object Storage. The students at Korea University asked excellent questions about the existing architecture and design; I anticipate seeing future research for OpenStack coming from Korea University in the near term.

Wednesday Feb 23 – OpenStack User Group Launch Seminar

The official OpenStack Korea User Group launch was attended by 120+ at the Coex Conference Center in Seoul. Speakers included Stephen Spector, John Willis (OpsCode), Paul Guth (Cloudscaling.com), Jaesuk Ahn (KT), Jinkyung Hwang (KT), Kyungyoon Kim (Clunet), Donghoon Kim (KT), Jaeuk Kang (FeelingK), Bundo Kang (Ubuntu Korean Community), and Byung-gon Kim (Java Community). The event detailed OpenStack, its role in the open source community, and ways for the Korean technical community to become active members with OpenStack. The event also coincided with the launch of the new Korean OpenStack User Group website at http://www.openstack.or.kr/.

KT has done an excellent job of evangelizing OpenStack within Korea and the sizable audience will lead to increased OpenStack efforts in Korea. Korea is on its way to being a leader in the development and promotion of OpenStack within our global community.

Tweet of the event from Jaesuk Ahn:

@songerie: I mean it. It’s packed. 🙂 http://yfrog.com/gy6y6nj #openstack korea community opening seminar.

Thursday Feb 24 – OpenStack at National IT Industry Promotion Agency

KT once again took the lead in promoting OpenStack at a government open source conference http://www.oss.kr/oss_etc1/2501 by presenting an overview of the project and how the government can engage with our open source community. As this talk was done in Korean (and I am more than a little rusty), I passed the baton to Jaesuk Ahn from KT who is an outstanding community leader in OpenStack and is driving our project to both business and government agencies throughout South Korea.

Final thoughts on my week in Korea:

  • OpenStack is a serious competitor for cloud computing in Korea and there is a high degree of interest to learn more about our project across both business and government agencies
  • KT, Jaesuk Ahn and his team are fantastic ambassadors for our open source community in Korea and their dedication to the success of OpenStack is evident in both their actions and passion for OpenStack. I leave Korea confident that our community has a great team in place to spread the OpenStack message in Korea
  • I anticipate one or two more events for OpenStack in Korea this year and hope to see more OpenStack community members add Korea to their 2011 travel plans as OpenStack is happening here
  • I encourage anyone wanting to learn how to promote OpenStack in their country to contact Jaesuk and his KT team; they have much to teach the rest of us in their ability to drive the open source message

Finally, thank you to both Jaesuk Ahn from KT, his team (cloud team at project expert group, KT) – especially Yunbum Huh who did all the “behind the scenes work”, as well as Professor Chuck Yoo for being outstanding hosts this week. With their support, we were able to promote OpenStack to the business, academic, and government communities in four days. Not bad for a weeks work.


Tags:

Community Weekly Update – Take 2

OpenStack Community Newsletter – February 18, 2011

This weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please email [email protected].

EVENTS

DEVELOPER COMMUNITY

GENERAL COMMUNITY

COMMUNITY STATISTICS (2/11– 2/17)

  • Data Tracking Graphs – http://wiki.openstack.org/WeeklyNewsletter
  • OpenStack Compute (NOVA) Data
    • 28 Active Reviews
    • 166 Active Branches – owned by 44 people & 9 teams
    • 1,398 commits by 56 people in last month
  • OpenStack Object Storage (SWIFT) Data
    • 4 Active Reviews
    • 49 Active Branches – owned by 19 people &  3 teams
    • 204 commits by 12 people in last month
  • Twitter Stats for Week:  #openstack  tweets;  re-tweets; all OpenStack total tweets  ** Looking for new tool to gather this data **
  • Bugs Stats for Week:  203 Tracked Bugs; 58 New Bugs; 31 In-process Bugs; 5 Critical Bugs; 18 High Importance Bugs; 52 Bugs (Fix Committed)
  • Blueprints Stats for Week:  162 Blueprints; 6 Essential, 15 High, 24 Medium,  21 Low,  96 Undefined
  • OpenStack Website Stats for Week: 9,339 Visits, 20,657 Pageviews, 53.59% New Visits
    • Top 5 Pages: Home 43.18%; /projects 11.82%; /projects/compute 17.42%; /projects/storage 12.59%; /Community 6.72%

OPENSTACK IN THE NEWS

Tags:

Community Weekly Update (Feb 11 – 18)

My apologies to the community but I was unable to complete the newsletter today as I was at an event in San Antonio, TX at a local college that did not have Wireless Internet, hard to believe but it is true – not all colleges have wireless. Thus, I am delayed in putting out the weekly newsletter until Monday morning as I am traveling this weekend. I will grab the weekly data for tracking and have that information also updated on Monday.

As for next week, I will be in South Korea for the launch of the OpenStack Korea User Group sponsored by KT and doing various other presentations on OpenStack at Korea University and the National IT Industry Promotion Agency. I will be sure to post lots of pictures from these events on the OpenStack Flickr account.

Tags: