Tag: OpenStack


Boris Renski of Mirantis presents: What’s new in OpenStack Folsom | Webcast 4 October 2012

October 2nd, 2012 — 6:34pm

Date: Thursday 4 October 2012 Time: 9am PT/12 Noon ET/6pm CET
Sign up here.

As many of you know, the Folsom release marks OpenStack’s transition from a service provider platform to an enterprise ready solution, with its baseline features set and hardening for enterprise usage in place.

I’d like to invite you to join me and Piotr Siwczak, Senior Staff Engineer at Mirantis and contributor to OpenStack, for technical overview of what’s new in the Folsom Release this Thursday, October 4th, at 9am Pacific.

Here’s what we’ll cover.

  • Synopsis of market developments since the April Essex release
  • New capabilities and user features: Nova, Cinder, Keystone, hypervisor support
  • Quantum and Load Balancer as a Service
  • Under-the-covers with key new architectural features
  • Q&A

The webcast is targeted both to experienced OpenStack users and cloud infrastructure teams considering new deployments. Click here for details on signing up.  Seats are limited.

Date: Thursday 4 October 2012
Time: 9am PT/12 Noon ET/6pm CET
You can review the Mirantis Privacy Policy here.  

Boris Renski, EVP and co-founder of Mirantis, is a member of the OpenStack Foundation Board.

 

Comment » | Event, Webinar

Australian OpenStack User Group events this month

August 3rd, 2012 — 3:23am

A couple of upcoming Australian OpenStack User Group (AOSUG) events I want to tell everyone about…

Hobart August 17 – OpenStack Chef and Puppet Code Dojo, the day before PyConAU!

PyCon Australia is the national conference for users of the Python Programming Language and is being held this year on August 18 and 19. This third PyConAU conference is being held in the beautiful city of Hobart in Tasmania. There’s a fantastic weekend lined up, including two days of fun and informative sessions on every aspect of the Python ecosystem. There’s also exciting social events planned, including the CodeWars programming tournament, the Women in Python breakfast and a conference dinner cruise and reception at Peppermint Bay.
PyConAU Registration is here.

On the preceding Friday the 17th we’re holding an OpenStack Code Dojo featuring Chef and Puppet. Special Guests for morning talks (live via satellite, well sorta, I’ve just always wanted to say that), will be Matt Ray from Opscode and Dan Bode from Puppetlabs. This will be followed by an afternoon code dojo specifically geared around creating, managing and versioning virtual machines. Food and drinks will be provided, my thanks to David Flanders, the PyConAU team and NeCTAR for their help organising this. Aptira will be throwing on the beverages for an after party of sorts! To participate in the day please go to the aosug.openstack.org.au site and RSVP.

Adelaide August 28 – Meetup

In conjunction with SAGE-AU and many thanks to Robert Mibus for the invitation, we’re inviting South Australian OzStackers to join the regular Adelaide chapter of SAGE-AU’s monthly meeting in Adelaide on August 28. Numbers for this event are limited so if you’re an SA OzStacker please jump onto http://aosug.openstack.org.au and RSVP to reserve your spot.

September events are planned for Melbourne and Sydney, and Brisbane in October.

Anyway that’s all from me for now, hope you all have a great Friday.

Cheers
Tristan
Australian OpenStack User Group

Comment » | community, Event, Meetup

Here is what happens inside Nova when you provision a VM

May 15th, 2012 — 2:06pm

At the Essex conference summit this past month, we presented a session on  OpenStack Essex architecture. As a part of that workshop we visually demonstrated the request flow for provisioning a VM and went over Essex arthicture. There was a lot of interest in this material; it’s now posted in Slideshare:

In fact, we’ve packaged up the architecture survey/overview as part of our 2-day Bootcamp for OpenStack. The next session is scheduled 14-15 June. This time around will carry out the training at the Santa Clara CA offices of our friends at Nexenta. Last course was delivered at our Mountain View office right before the OpenStack summit in April to a sold out crowd. You can find more information about the course at www.mirantis.com/training

1 comment » | Documentation, Event, Partner, Uncategorized

I hear the Essex Train a-coming

March 31st, 2012 — 5:39am

With Essex train in the wilds of testing, and the Essex release intended date less than 10 days away, we are pretty excited about everyone descending on San Francisco — practically our home town — for the Design Summit and Conference.

Here at Mirantis, the company famous across OpenStack community for distributing vodka bottles at OpenStack meetups, we are gearing up in a big way for the summit and conference. If you haven’t seen the agenda, here’s what we’ve got teed up:

(1) We’ll start the frenzy with Just-in-time-Training: we have a few seats left at our 2-day OpenStack Boot Camp, crammed into the weekend of April 14-15, right before the summit and conference. REGISTER HERE and come to the event fully prepared to torment speakers and presenters with insidious technical questions about OpenStack technology and its future.

(2) Our team will participate in / moderate a few exciting sessions during the conference: OpenStack and Block Storage, OpenStack and High Performance Computing, Expanding the Community. Please be sure to pay us a visit.

(3) …and just to show how happy we are to have you here, we invite everyone at the conference to join Mirantis Summit Kick-Off Party. This is how we party at Mirantis! Vodka bottles and fun times in the best traditions of all our events are guaranteed. Be sure not to miss.

Looking forward to receiving everyone at the 2012 OpenStack Design Summit and Conference.

Comment » | community, Event, Partner

OpenStack Sydney Meetup Special Event with James Williams

February 28th, 2012 — 4:17am

Tristan Goode from Aptira, James Williams from NASA, and Phil Rogers from Aptira

Last Friday February 24 at the Harbour View Hotel in Sydney the Australian OpenStack User Group had a special meetup with James Williams, CIO of NASA AMES Research Centre. For those not in the know James and AMES had a key role in the establishment of Nebula and OpenStack.

Last week we heard James was in the country for a conference and was passing through Sydney, so an invitation to meet the User Group was extended and James graciously accepted at very short notice. James’ enthusiastic evangelism for OpenStack was very inspiring and everyone attending managed to get a chat in. On behalf of the Australian OpenStack User Group and the evening’s sponsor Aptira, thanks to James for fitting us into his hectic schedule.

More pictures are available over at TechWorld.

Head to our Australian Meetup group to get involved, or join the AU Google group.

 

Comment » | Event, Meetup

Under the hood of Swift: the Ring

February 15th, 2012 — 3:45am

This is the first post in series that summarizes our analysis of Swift architecture. We’ve tried to highlight some points that are not clear enough in the official documentation. Our primary base was an in-depth look into the source code.

The following material applies to version 1.4.6 of Swift.

The Ring is the vital part of Swift architecture. This half database, half configuration file keeps track of where all data resides in the cluster. For each possible path to any stored entity in the cluster, the Ring points to the particular device on the particular physical node.

There are three types of entities that Swift recognizes: accounts, containers and objects. Each type has the ring of its own, but all three rings are put up the same way. Swift services use the same source code to create and query all three rings. Two Swift classes are responsible for this tasks: RingBuilder and Ring respectively.

Ring data structure

Every Ring of three in Swift is the structure that consists of 3 elements:

  • a list of devices in the cluster, also known as devs in the Ring class;
  • a list of lists of devices ids indicating partition to data assignments, stored in variable named _replica2part2dev_id;
  • an integer number of bits to shift an MD5-hashed path to the account/container/object to calculate the partition index for the hash (partition shift value, part_shift).
List of devices

A list of devices includes all storage devices (disks) known to the ring. Each element of this list is a dictionary of the following structure:

Key Type Value
id integer Index of the devices list
zone integer Zone the device resides in
weight float The relative weight of the device to the other devices in the ring
ip string IP address of server containing the device
port integer TCP port the server uses to serve requests for the device
device string Disk name of the device in the host system, e.g. sda1. It is used to identify disk mount point under /srv/node on the host system
meta string General-use field for storing arbitrary information about the device. Not used by servers directly

Some device management can be performed using values in the list. First, for the removed devices, the 'id' value is set to 'None'. Device IDs are generally not reused. Second, setting 'weight' to 0.0 disables the device temporarily, as no partitions will be assigned to that device.

Partitions assignment list

This data structure is a list of N elements, where N is the replica count for the cluster. The default replica count is 3. Each element of partitions assignment list is an array('H'), or Python compact efficient array of short unsigned integer values. These values are actually index into a list of devices (see previous section). So, each array('H') in the partitions assignment list represents mapping partitions to devices ID.

The ring takes a configurable number of bits from a path’s MD5 hash and converts it to the long integer number. This number is used as an index into the array('H'). This index points to the array element that designates an ID of the device to which the partition is mapped. Number of bits kept from the hash is known as the partition power, and 2 to the partition power indicates the partition count.

For a given partition number, each replica’s device will not be in the same zone as any other replica’s device. Zones can be used to group devices based on physical locations, power separations, network separations, or any other attribute that could make multiple replicas unavailable at the same time.

Partition Shift Value

This is the number of bits taken from MD5 hash of '/account/[container/[object]]' path to calculate partition index for the path. Partition index is calculated by translating binary portion of hash into integer number.

Ring operation

The structure described above is stored as pickled (see Python pickle) and gzipped (see Python gzip.GzipFile) file. There are three files, one per ring, and usually their names are:

account.ring.gzcontainer.ring.gzobject.ring.gz

These files must exist in /etc/swift directory on every Swift cluster node, both Proxy and Storage, as services on all these nodes use it to locate entities in cluster. Moreover, ring files on all nodes in the cluster must have the same contents, so cluster can function properly.

There are no internal Swift mechanisms that can guarantee that the ring is consistent, i.e. gzip file is not corrupt and can be read. Swift services have no way to tell if all nodes have the same version of rings. Maintenance of ring files is administrator’s responsibility. These tasks can be automated by means external to the Swift itself, of course.

The Ring allows any Swift service to identify which Storage node to query for the particular storage entity. Method Ring.get_nodes(account, container=None, obj=None) is used for identification of target Storage node for the given path (/account[/container[/object]]). It returns the tuple of partition and dictionary of nodes. The partition is used for constructing the local path to object file or account/container database. Nodes dictionary elements have the same structure as the devices in list of devices (see above).

Ring management

Swift services can not change the Ring. Ring is managed by swift-ring-builder script. When new Ring is created, first administrator should specify builder file and main parameter of the Ring: partition power (or partition shift value), number of replicas of each partition in cluster, and the time in hours before a specific partition can be moved in succession:

swift-ring-builder <builder_file> create <part_power> <replicas> <min_part_hours>

When the temporary builder file structure is created, administrator should add devices to the Ring. For each device, required values are zone number, IP address of the Storage node, port on which server is listening, device name (e.g. sdb1), optional device meta-data (e.g., model name, installation date or anything else) and device weight:

swift-ring-builder <builder_file> add z<zone>-<ip>:<port>/<device_name>_<meta> <weight>

Device weight is used to distribute partitions between the devices. More the device weight, more partitions are going to be assigned to that device. Recommended initial approach is to use the same size devices across the cluster and set weight 100.0 to each device. For devices added later, weight should be proportional to the capacity. At this point, all devices that will initially be in the cluster, should be added to the Ring. Consistency of the builder file can be verified before creating actual Ring file:

swift-ring-builder <builder_file>

In case of successful verification, the next step is to distribute partitions between devices and create actual Ring file. It is called ‘rebalance’ the Ring. This process is designed to move as few partitions as possible to minimize the data exchange between nodes, so it is important that all necessary changes to the Ring are made before rebalancing it:

swift-ring-builder <builder_file> rebalance

The whole procedure must be repeated for all three rings: account, container and object. The resulting .ring.gz files should be pushed to all nodes in cluster. Builder files are also needed for the future changes to rings, so they should be backed up and kept in safe place. One of approaches is to put them to the Swift storage as ordinary objects.

Physical disk usage

Partition is essentially the block of data stored in the cluster. This does not mean, however, that disk usage is constant for all partitions. Distribution of objects between the partitions is based on the object path hash, not the object size or other parameters. Objects are not partitioned, which means that an object is kept as a single file in storage node file system (except very large objects, greater than 5Gb, which can be uploaded in segments – see the Swift documentation).

The partition mapped to the storage device is actually a directory in structure under /srv/node/<dev_name>. The disk space used by this directory may vary from partition to partition, depending on size of objects that have been placed to this partition by mapping hash of object path to the Ring.

In conclusion it should be said that the Swift Ring is a beautiful structure, though it lacks a degree of automation and synchronization between nodes. I’m going to write about how to solve these problems in one of the following posts.

More information

More information about Swift Ring can be found in following sources:
Official Swift documentation – base source for description of data structure
Swift Ring source code on Github – code base of Ring and RingBuilder Swift classes.
Blog of Chmouel Boudjnah – contains useful Swift hints

Comment » | Documentation

OpenStack Jenkins dashboard available for testing Ubuntu snapshots

January 27th, 2012 — 7:50am

The keener eyed of you may have noticed:

https://jenkins.qa.ubuntu.com/view/Precise%20OpenStack%20Testing/

James Page has setup the jobs in the Ubuntu OpenStack QA Lab to start publishing to the public Jenkins QA instance this morning. We now have automated build testing of all core OpenStack components triggered from upstream trunk commits. This is followed by automated deployment (-deploy) of OpenStack in the lab with a serving of testing (-test) once its all up and running.

Credit to Adam Gandelman for the Juju charm work, deployment framework and test execution and to Chuck Short for the hugely misnamed tarball.sh script which completes the git/bzr/packaging fu to build and deploy OpenStack packages!

The plan is to get the upstream Tempest test suite running in the lab; at the moment we are running a more limited test script just to ensure that you can spin up and instance and see it on the network.

(Crossposted from cloud.ubuntu.com)

1 comment » | Development

Great Turnout at the first Austin OpenStack Meetup

October 30th, 2011 — 11:01pm

.

Last Thursday, a number of us Austin OpenStack fans decided to get together and talk OpenStack, Diablo, Crowbar, and more.

We had a fantastic turnout of almost SEVENTY people who came out that night – almost at near capacity for our venue, Tech Ranch Austin.  A number of startups where represented, as well as a number of notable OpenStack partners like Rackspace, Canonical, and Dell (the company I work for), who sponsored this first OpenStack meet up in Austin.

This meetup coincided with the Rackspace Cloud Builders OpenStack training, being held at the Dell campus that entire week, so a number of OpenStack students from that class, many who had flown in for class from out of town / state, were able to make it as well.

It was a great pleasure for us here at Dell to sponsor the first Austin meetup for OpenStack, and I look forward to our community growing as other partners help us sponsor future meet ups.

You can get more details on what was discussed at the meetup at Rob’s blog – www.RobHirschfeld.com.

If you’re in the Austin area, and are interested in joining the OpenStack Austin meetup group, join us at www.meetup.com/OpenStack-Austin.

1 comment » | community, Meetup, Partner

OpenStack Birthday Videos (continued)

July 27th, 2011 — 11:52pm

More participating companies celebrating OpenStack’s first birthday.

MemSet 1 Year Birthday OSCON from OpenStack on Vimeo.

Cybera OpenStack 1 Year Birthday from OpenStack on Vimeo.

Zadara Storage Happy Birthday to OpenStack at OSCON from OpenStack on Vimeo.

1 comment » | community, Event, Video

OpenStack Day in London Recap

July 18th, 2011 — 10:41am

On Wednesday we held an OpenStack Day in London — the first for our community in Europe.  It’s very obvious we should have done this much sooner.  We never know how many people to expect at these events, and had planned for 125 or so.  In the end, about 350 people attended.  The catering held up well to the surge in attendees, but the wifi didn’t fare so well.  We will adjust going forward.  Here are some of my observations from the day:

1.  Not only was a substantial portion of the audience very familiar with OpenStack, many had already deployed it.  I met individuals and companies from around Europe who have already deployed OpenStack clouds.  Because we don’t track software installations, we have little idea how many of the 35,000+ downloads from Launchpad are actually running.  However, identifying yourself in the community is very critical as it allows us to help tell your story, helps other users get information, and in general helps move OpenStack forward.  So if you have deployed OpenStack, please let us know by contacting Stephen Spector (stephen.spector@openstack.org).  We are putting togther case studies and if you participate we will get you some free OpenStack schwag!

2.  Many of the attendees had traveled from other parts of Europe to attend including the France, Hungary, Norway, Finland and Spain.  We are working hard to establish user groups around the world so that anyone wanting to learn about OpenStack or chat about the project with peers will have the chance to do so.  If you are interested in running one, please also contact Stephen.

3.  The ecosystem is rapidly maturing with existing members increasing their investments, new ones continuing to join, and new businesses getting funded around OpenStack.  In particular, the discussion about OpenStack distributions from companies such as Citrix, Canonical and StackOps indicates how companies are investing in this technology as the right cloud solution for their customers.  Citrix highlighted how their acquisition of Cloud.com was a doubling down of their commitment to the project.  All of this activity bodes well for the creation of a broad range of OpenStack solutions for customers.

4.  We have some smart developers, system administrators and operators working on this project.  Vish Ishaya, Jonathan Bryce, Josh McKenty and Chmouel Boudjnah all gave excellent overviews of the technology.  The audience questions and hallway conversation also indicated that our flock of community members is well above average.

An installfest was held over pints right after the event.  We filled up the one room we booked and had to take a second.  There was some good community bonding going on for hours.  Lots of discussion was also had about a future event in Europe, so we have become looking at Paris for September.  Stay tuned for more details.

I want to thank Canonical, Equinix, Dell, Citrix, Rightscale and Rackspace for their sponsorship of the event.  If you have feedback for us, don’t hesitate to drop me a line at jim@openstack.org or @jimcurry on Twitter.

1 comment » | community, Event, Meetup

Back to top