The State of OpenStack…and What’s Next?! (Part One)

 

September 10, 2010

Wow, what a crazy month it has been! We’re really excited by all the buzz and activity that’s going on in OpenStack–in IRC (freenode.net #openstack) and on the mailing lists (launchpad.net/openstack), as well as on Twitter (@openstack) and in the media. But hype time is over and it’s time for us to begin delivering on the promises of OpenStack.

OpenStack Compute and Object Storage already have a number of important features in the code base today including:

  • managing virtual machines
  • network configuration for those VMs
  • a fully distributed, replicated storage system
  • storing, retrieving, and replicating objects and files up to 5GB in size

In addition, we have already released interfaces for controlling an OpenStack environment via a web control panel, iPad, iPhone or Android device.

But there is more coming. Chief Architect Rick Clark recently sent out an update to the community outlining the plans for our upcoming October 21 “Austin” release. This initial release of Compute will advance the project a long way, in that it will allow the community to deploy proofs of concept on hundreds of servers on a single cluster and begin developing on the platform.  This is an important milestone on the path to reaching the long term goals of the project, and will be followed by new releases every 3 months.  We will discuss the follow on releases in another blog post, and in depth at the November 9-12th Design Conference in San Antonio.

There are some important changes and updates in the Austin release that are worthy of further detail.  Let me start with what we know will be available on October 21:

1.  The OpenStack API

The Austin release will include the official OpenStack RESTful API, which initially is based on the existing Rackspace Cloud API (published under Creative Commons in 2009).   It will also include additional functionality such as role-based access controls and additional networking actions. This API will be the official OpenStack API and it will evolve with the platform and needs of the community.

What about support for alternative APIs, such as EC2?  The EC2 compatible API, already in the code base today, will remain and be maintained; however, it is important for the project to have an official API that is tied directly to the OpenStack roadmap and feature set. We want to ensure that future OpenStack innovation can be driven by the community and not be restricted to the functionality of outside cloud APIs. The sub-projects are built in a way that will allow multiple APIs to be supported, so if there is an existing API that is really important to you (or one that comes along in the future), it is possible for you to add in support for that as well.

2.  Hypervisor & Image Support

A key tenant of OpenStack is hypervisor neutrality.  OpenStack currently uses libvirt, which provides an abstraction level for hypervisor management for a number of different hypervisors. The Austin release will contain support for XenServer, KVM, and UML. It also supports VirtualBox, which allows people to launch virtual machines on their laptop, making testing and development easier for community members.

For booting up new virtual machines, OpenStack will make use of the Rackspace Cloud’s imaging framework. The framework is based on a tarball of a bootable filesystem.  We will have more detail on future image support direction soon.

3.  Unifying Compute and Object Storage

Its important that the sub-projects of OpenStack can be utilized independently, but we also want to maximize the interfaces to make the entire system hum.  To that end, the authentication system for both Compute and Object Storage will be unified. We are also making Object Storage available as the image store for Compute servers.

4.  Networking Model

We will be supporting two network models in OpenStack: statically assigned, real Internet IP addresses; and private IP addresses within a dedicated subnet, connected via NATing from a private VPN to the public internet. The API will allow users to choose which model they want. Role-based access controls for firewalling will also be added.

These are features that are already in progress and we are comfortable will be complete for the Austin release. But we would like to get more in! We received a lot of input during the Design Summit in July–and continue to see suggestions roll in from the community. With the community’s help, we would like to get these features for the Austin release as well:

1.  Better Server Volumes

The existing Rackspace Cloud supports several great features for server volume management, and we want to get those into OpenStack as soon as possible: resizing servers, snapshotting volumes, and “Rescue Mode”.

2.  Better Server Management

The existing Rackspace Cloud also has more sophisticated server management tools, which we’re actively porting into OpenStack: “Rescue Mode”, and Web-based console access.

3.  Underlying (Core) Refactoring

We want to have as many people running, working on, and contributing to OpenStack as possible. Pragmatically, that means we need to try and make the codebase as friendly as possible. Current efforts are on three broad tracks:

a.  Packaging, deployment recipes, and installers

b.  Making the programming models as standard and understandable as possible

c.  Abstracting the datastore from the object model (enabling folks to use SQL or alternate KVS systems)

d.  Cleaning up code and providing consistent, useful documentation (read about our new tech writer)

We think these last three items are important and urgent, and hope to get active community engagement on their development. Hopefully they will make it into the Austin release, although since this is a time-boxed release, we can’t be certain.  Our priority will be to deliver a stable set of features rather than broad. If they are not ready by mid-October, we expect the community will be able to deliver them by our next release in January 2011 (more on that later).

Thanks Stackers for getting this project off to a great start!  Please reach out if you have any feedback or need any assistance.

On behalf of the OpenStack Team and Community,

Jim Curry
Chief Stacker
@jimcurry, [email protected]

The Second OpenStack Design Conference

 

September 9, 2010

The next OpenStack Design Conference is coming up fast, and we are excited to get everyone back together who attended the conference in July, and welcome a lot of new faces who’ve joined the community since the launch at OSCON.  Since the conference is just a few short weeks away, we wanted to let everyone know the moment we’d locked on a date and secured a location:

Dates:  November 9th-12th

Location:  The Weston Centre in Downtown San Antonio, TX

Address:  112 E. Pecan St. San Antonio, TX 78205

Map with nearby hotels:


View Larger Map

While we’re still working on the Agenda, the format will be the same as the last conference. Namely, the first two days (November 9th-10th) will be filled with different tracks (business and technical) with a lot of active discussion on goals and design decisions for the project. The last two days (November 11th-12th) will be used for hacking on the code and finalizing any design decisions for the next release that require prototyping / live testing. At the last summit, the heaviest attendance was the first day (when we have a general session), a bit less the second day, then about half on the final two days as the focus turns to coding. We are working on some evening plans as well.

If you can only make it one day, come on the first day November 9th.

What’s next?

1)  Hotel Block:  We are looking into the possibility of getting a block of rooms with a discount.  Given the short timeframe and the fact that other conferences are going on that week, you may want to go ahead and just book one of the hotels above, all of which are walking distance to Weston Centre.

2)  Invites:  We’ll be sending the formal invites via email to the registration process as soon as possible.  To ensure you receive the registration link, please sign up below:

3)  Preparing for the conference!  We will need your help as we devise schedules and work on the logistics.  Volunteers needed!

What to do now:

1)  Forward this post to others in the community, tweet, re-tweet rinse and repeat

2)  Book Travel

3)  As always, follow @openstack on twitter for the latest news and updates

4)  Sign up below to get the registration link when it’s ready

Notify Me When I Can Register
* indicates required

Mark Collier
Chief Community Stacker
@sparkycollier

Content Stacker, Reporting for Duty

 

September 7, 2010

Well, hi there. Glad you could make it. Let me introduce myself. I’m Anne Gentle, the newest addition to the hardworking OpenStack team. And I’m so eager to get started I can barely contain myself. I’ve been in lurker mode for a few weeks and now it’s time to reveal my role – I’m the OpenStack Technical Writer.

How did I get here? I first started working in open source a few years ago, and I applied my technical communication skills gladly because there is often a gap in documentation and support in open source projects. I became a big supporter of FLOSS Manuals, where we write free manuals for free software. We work in a wiki and in Booki, a new collaborative authoring platform. We also employ a book sprint technique where we focus collaborative authoring efforts into week-long sprints. I’ve helped with a few book sprints and think it’s a great technique for open documentation.

Where do I go from here? Today I want to start outlining some documentation strategy to get your feedback and tweak it. We’ll be testing doc efforts as we go and likely start with the Object Storage area.

Audit and Analysis

This phase should answer questions like, what exists already and how is it doing? I plan to do a thorough content audit right away. Seems like there are a lot of places to write things down, and I want to determine what’s out there. I’ve started reading through the Swift documentation and learning about Sphinx. We’ve all been around technology long enough to know that tools are not the complete solution, that makers and crafters help the content go from adequate to exceptional.

I’ll also study other projects (I’ve been observing several different projects for a while now). I’d like to hear your preferences here as well – how do you get information from similar projects?

An audience analysis also comes in this phase – who needs the documentation and what tasks do they accomplish with it? I expect to be prioritizing both administrator documentation and developer documentation as well as introductory information for those on their initial approach to OpenStack.

Actions:

  • Build a table on the wiki showing where content exists.
  • Look for reuse opportunities as well as unifying characteristics such as audience, level of difficulty, chunking, and organization of content.
  • Describe the typical readers and the tasks they need to accomplish.

Creation and Collaboration

After an analysis phase, where I will avoid analysis paralysis, I want to look at content structure – what types of information should be created and where can you find them? Information typing is a huge part of what makes my life easier and more organized as a writer. Jacob Kaplan-Moss wrote a great post about writing effective documentation detailingwhat to write. I completely agree that you can write any technical documentation with tutorials, topical guides, and reference guides. I believe this approach will help us write helpful, useful documentation. I would like to build in documentation as part of the ethos of the community, part of what we value.

At first glance, the types of documentation needed may be easily divided by reader – developer documentation (which could live in the Python code with Sphinx, in the specs in Launchpad, and in Etherpad when real-time collaboration is needed) and administrator documentation (which could live on the wiki, that’s yet to be determined). We always want to consider translation and local community support, because not speaking English as a first language shouldn’t deter you from contributing to the documentation.

Actions:

  • Create a Launchpad for documentation for requests and tracking purposes.
  • Create a starter style guide for the OpenStack wiki.
  • Talk to interested parties about contributing.
  • Set up starter templates on the wiki.
  • Write and edit as much as possible.
  • Plan for doc sprints on specific topics to coincide with in-person events.

Goals

I think the documentation should help support users first and foremost. To me, pragmatic, viewable results come first, fancy-pants solutions are second. I think that philosophy works great at Rackspace and I’m pleased to see it’s a core value – Results first, substance over flash.

How about you? What do you think should be the goals of OpenStack technical documentation? Support? Adoption? Education? All of the above? Let us know.

What can you do to help with doc?

Please, introduce me to people who want good documentation and recognize good doc.  Introduce yourself or name others. Send me examples of sites that do it well. Write things down as you go, even if they are rough notes. Help us shape a doc strategy so it fits our community.

You can read more about me on my blog at JustWriteClick.com, on Twitter @annegentle, and get in touch any number of ways – comment on my posts, talk to me on Twitter, email me at anne at openstack dot org.

I’ll be asking lots of questions as I get started, but these are my initial thoughts. Ready to start stacking!

Chef Meets the iPad with OpenStack

 

August 26, 2010

Since OpenStack was announced, we’ve seen a flurry of activity in the developer community. Hundreds of people have made over a thousand contributions and we couldn’t be more excited.

Today I’d like to share one of the more unconventional contributions with you. Most of the development activity in OpenStack is in the storage and compute systems, but we also have front end projects for people who use OpenStack and need an easy way to manage their cloud resources.

One of these projects is the OpenStack iPad app, which is based on the Rackspace Cloud Pro iPad app that is currently in the App Store. This app uses the OpenStack compute and storage APIs to help you manage your cloud resources, and offers a few features outside of the scope of the APIs, such as viewing RSS system status feeds, pinging your compute nodes from several locations around the world, and emailing files from OpenStack Object Storage.

Our newest feature, made possible by working with people from Opscode, is integration with Chef and the Opscode Platform. Before, when you created a compute node on the iPad, you would then need to log into the node and manually install any software, data, and configurations you may need. This could take a considerable amount of time and effort for some deployments, but not anymore. If you’re a Chef user, you can configure the iPad app with your Chef or Opscode credentials to automatically run any recipes or role assignments immediately after the node is provisioned. Look ma, no shell!

Let’s suppose you’re running a Cassandra cluster to horizontally scale your data, and you’re out of town on a business trip. Your laptop is in your hotel room, and you’re stuck in a conference room across town with only your iPad. Then, your monitoring service sends you an SMS telling you that all of your Cassandra machines are running at capacity. But you were smart; you used Chef to automate your infrastructure by creating roles for all of your compute nodes. You simply log into the iPad app and provision a new node with “role[cassandra]” as your run list. The machine boots up and adds itself to your Cassandra cluster without requiring anything else from you. Awesome!

I’ve included a video below showing you the entire process as I bootstrap a compute node running Apache 2 and MySQL. I hope you enjoy it, and if you have any questions or want to talk about contributing to the OpenStack iOS apps, get in touch with me via email at [email protected], Twitter at @greenisus, or on the phone at 901-299-9306. Even if you’re new to Objective-C and Cocoa, I’ll be happy to help you get started and learn everything you need to know to help make the OpenStack apps even better.

OpenStack Community Update

 

August 16, 2010

It’s been four weeks since the launch of OpenStack, and we’ve been amazed by the response. A huge thank you to the developers who have been working tirelessly to move the OpenStack Object Storage and Compute projects along at a rapid pace, and to the commercial supporters who have been rallying around an open source cloud alternative. Since the launch of the project, we’ve reached some significant milestones and wanted to share the latest:

  • OpenStack Compute now supports three hypervisors (in just three weeks!) — KVM, XenServer (thanks Ewan Mellor @citrix) and VirtualBox (thanks Justin Santa Barbara ) furthering the community’s goal of broad support
  • Between the two projects, there have been 1,250 code commits and there are more than 55 active branches
  • To date more than 35 bugs have been fixed. The first bug was reported within 8 minutes of launch, and a patch submitted within 20!
  • There has been an average of 130-140 developers in the IRC channel each day, a strong showing for a relatively new community
  • More than 30 companies have joined the OpenStack community, and several are hiring. New members include Cloud Central, Morphlabs, and MidoKura. Check out our jobs page or submit your own.

We’re making progress. How do you get involved?

  • Join our IRC channel #openstack on freenode
  • Find the code and community tools hosted on Launchpad.
  • Ready to contribute code?  complete the process here then checkout our guide to Life with BZR and Launchpad
  • Once you’re part of Launchpad, join the mailing lists (links are inside the Launchpad pages for “Nova” and “Swift” which are the OpenStack Compute and Object Storage projects, respectively)
  • Attend upcoming meetings (see below) and our next design conference later this year; stay tuned for the dates
  • Follow @openstack on twitter — where we’ll be communicating project updates and upcoming events
  • Host your own Stackup!  Tell us about it and we’ll be sure to spread the word.

Upcoming Stackups

  • The Silicon Valley Cloud Computing Group is hosting their first Stackup on Tuesday August 24th.  Over 160 people have already RSVPd, so if you’re in the area sign up now.
  • Thursday August 26th there will be a Stackup at the NASA Ames Research Center with a focus on “Swift” the Object Storage project.  Details here.

Mark Collier

VP of Community & Business Development

OpenStack

@sparkycollier

You Are Now Free to Move About in the Cloud

 

July 22, 2010

Today Cloud.com announced that we have joined the OpenStack Initiative, providing our commitment to support, collaborate and contribute to the success of this community and it’s efforts to drive open standards and innovation in the cloud computing market.

We are very excited to be part of this effort as it signifies a few significant tipping points in the industry, and share Rightscale’s view that this is a game changing event. Linux and many open source projects reached their curve in the “hockey stick” of growth only when the customers joined in the effort of advancing the market. On top of Rackspace and NASA, I think it speaks volumes to see the OpenStack initiative include NTT, Peer1, Limelight, Softlayer and others all joining together as part of this initiative. Think about that for a second. These are all competitors joining together to advance the market.

Back to Cloud.com’s role in this initiative. When I talk about joining the cause, I want to call out three specific areas that we are already working on with the OpenStack team.

  1. Cloud.com will be adopting the OpenStack Object Storage platform into our CloudStack product, allowing us to immediately deliver a cloud storage offering to our open source IAAS stack. We’ve been approached by a number of service providers and enterprises across the board looking for a storage service that would deliver an Amazon equivalent storage architecture within their cloud. Their challenge? Vendors that they were relying on went bust leaving them without a solution or a trusted provider to enable them with this technology. The significance of Rackspace’s contribution of the OpenStack Object Storage is that Cloud.com can now take a proven, scalable, in production object storage and make that available to our customers. That’s pretty cool. We will, of course, also be contributing back to the project areas where we can improve implementation, management and orchestration of the service.
  2. Cloud.com will also be adding the OpenStack API implementation to our Common Cloud Framework, enabling customers and developers to leverage their existing cloud deployments. Interoperability is a key pinpoint for developers, customers and users and we are at a crucial point in the growth of this market where we need to truly enable cloud interoperability in order to achieve hyper growth.
  3. We will also be using our expertise in building out highly scalable, multi-tenant cloud environments for SPs and Enterprises to help advance the OpenStack initiative. We’ve been extremely successful in deploying large scale, in-production clouds and are looking forward to contributing towards commercializing and packaging the OpenStack for production use.

OpenStack and Rackspace/NASA have been working to solve certain, hard technology problems around scale and reliability in the cloud. Cloud.com and CloudStack will enable these technologies to work for an enterprise by integrating this technology with back office functionality, delivering interoperability through common cloud API’s and integrating additional user management and administrative tooling in a supported and sustainable offering. This makes partnering and working with OpenStack.org a no brainer decision for us.

Learn more about our OpenStack momentum and other musings of Cloud.com on twitter @clouddotcom or @ulander

Congrats to the OpenStack team. As I read in a tweet on the announcement (which I should have saved for proper attribution), this is just the bottom of the first inning … but open source has just brought some big hitters to the game.

Introducing OpenStack

 

July 19, 2010

If you were to pick one word to describe open source, it probably would be freedom. Freedom to innovate. Freedom to consume. Freedom to redefine. There is a great tradition of open source movements revolutionizing entire segments of the computing and software world by fostering freedom – Linux and operating systems, Apache and web servers, mySQL and databases — just to name a few. Those open source projects introduced freedom for both developers and consumers of technology, thus accelerating the pace of innovation and adoption. Today we are proud to add OpenStack to that list of revolutionary ideas, bringing a new era of freedom to the cloud marketplace.

What is OpenStack? Well, our mission statement says this:

To produce the ubiquitous OpenSource Cloud Computing platform that will meet the needs of public and private clouds regardless of size, by being simple to implement and massively scalable.

That is a pretty big ambition. The good news is that OpenStack is starting with code contributions from two organizations that know how to build and run massively scalable clouds – Rackspace and NASA. Rackspace has been in the cloud business for four years and now serves tens of thousands of customers on its cloud platform. Likewise, NASA began building their Nebula cloud platform two years ago to meet the needs of their scientific community.

Today, OpenStack consists of two projects. The first is a fully distributed object store based on Rackspace’s Cloud Files offering called "OpenStack Object Storage". The code is available today at OpenStack.org. The second piece is a scalable compute-provisioning engine based on the NASA Nebula cloud technology and Rackspace Cloud Servers offering called "OpenStack Compute." Developers can download components of OS Compute beginning today at OpenStack.org. The first release is expected to be available later this year. So starting today, anyone can build their own cloud using the same technology that underlies two of the largest and best ones out there.

But it’s not just code from Rackspace and NASA. Last week, more than 100 architects and developers from over 25 companies came to Austin to begin defining the roadmap for OpenStack… and more importantly to begin jamming away at the code! The list included developers from managed hosters, hardware and component manufacturers, enterprise software and service companies, other open source projects, and cloud tools vendors. These community founders are already driving our project forward, and we are actively seeking more contributors to do the same!

What does "openness" mean to us? "Open" and "open source" are thrown around a lot, so its worth specifically defining our commitment to the community:

  • COMMITMENT #1: We are producing truly open source software. No artificial limits will be placed or performance limitations maintained. No licensing model – one free, one paid – will be introduced. We are releasing the code under the Apache 2.0 license which allows the community to do with the software as they see fit, including implement into other distributions or “for fee” offerings.
  • COMMITMENT #2: We are committed to an open design process. Rackspace will provide dedicated project leads to guide the roadmap on behalf of the community. We will hold regular design summits—open to anyone—which will produce a roadmap to guide development.
  • COMMITMENT #3: All development will be done in the open. We will maintain a publicly available source code repository to simplify participation.
  • COMMITMENT #4: We will maintain an open community. Healthy, vibrant developer and user communities are the basis of any open source project. Most decisions will be made using a "lazy consensus" model. All processes will be documented, open and transparent.

You can learn more by going to the Rackspace press release. Also please follow us on Twitter (@openstack) for regular updates.

So the mission starts now! We are here to help you with this project any way we can, so please reach out and let us know what we can do. You can personally reach me anytime at [email protected] or by following me on twitter (@jimcurry).

Cloud freedom, open for business!

Jim Curry
OpenStack Lead, OpenStack.org