Tag: Developer


Community Weekly Review (Apr 20-27)

April 27th, 2012 — 4:50pm

OpenStack Community Newsletter – April 27, 2012

Welcome back to our regular publishing schedule. This week we still hear the echo of the Design Summit and Conference.

HIGHLIGHTS

Upcoming Events

Other news

Community Statistics

This week’s chart shows the geographical dispersion of participants to Folsom series of events in San Francisco. The information is derived from the work address provided by participants when they registered. Participants from USA were the large majority, around 70% of the over 1,000 participants, nonetheless it’s interesting to look at the distribution once the outlier is removed.

Participants to Folsom Design Summit and Conference, per nation (excluding USA)

This weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please leave a comment.

1 comment » | Communication, community, Newsletter

Automating Openstack Testing on Ubuntu

February 8th, 2012 — 8:44am

(Original Post)

During the Ubuntu precise development cycle the Canonical Platform Server Team have been working on automating testing of Openstack on Ubuntu.

The scope of this work was:

  1. Per-commit testing of Openstack trunk to evaluate the current state of the upstream codebase in-conjunction with the current packaging in Ubuntu precise and the current Juju charms to deploy Openstack.
  2. SRU testing for Openstack Diablo on Ubuntu 11.10.

Openstack do a lot of pre-commit testing through the use of gerrit with Jenkins; we wanted to supplement this with Ubuntu focused testing to provide another dimension to the testing already completed upstream.

So grab a coffee and make yourself comfortable; this is not a short read….

Lab Setup

The Ubuntu Openstack QA lab consists of 12 servers; the primary server in the solution is an Ubuntu 11.10 install providing the following functions:

  1. Juju – used to deploy Openstack charms in the Lab
  2. Cobbler to support server provisioning (using the Ubuntu Orchestra packages in Oneiric)
  3. Jenkins CI – provides triggering based on upstream commits to github repositories and general job control and reporting.
  4. Schroots for Oneiric and Precise for building packages locally
  5. A reprepro managed local archive for Oneiric and Precise
  6. Squid based archive caching to reduce installation times in the lab

This server also acts at the gateway into and out of the Lab (it’s setup as a NAT router).

The other 11 servers are registered in Cobbler; All servers are connected to a Sentry CDU (Cabinet Distribution Unit) which allows full power control from Cobbler – thanks goes to Andres Rodriguez for developing the required fence component for Cobbler to support this type of CDU.

Preseeded LVM Snapshot Installs

To initiate a new integration test run requires all machines to be powered down and re-provisioned from scratch.  It is essential that our deployment and test runs can cope the frequency of upstream commits, particularly as the frequency increases as Openstack approaches milestones and releases.   After getting the initial lab setup in place, we were able to tear down all machines, re-provision and deploy Openstack in ~30mins.

It was important that we are able to minimize the time taken to complete the testing cycle.   To do so, we’ve employed the use of LVM snapshotting and restoration of the root partition during the the netboot installation.   The process is as follows:

  1. Test run begins
  2. Juju deploys a service (i.e. nova-compute)
  3. A machine is netbooted and a preseeded LVM-based Ubuntu installation takes place onto /dev/qalab/root
  4. At the end of the installation, the root filesystem is moved to /dev/qalab/pristine-[release]-root and a snapshot created at /dev/qalab/root
  5. The machine reboots, runs Juju and deploys nova-compute as pat of the rest of the Openstack deployment. This deployment is smoke tested.
  6. The next test run begins.  All machines are terminated. Juju redeploys nova-compute, a machine is netbooted and Ubuntu installation kicks off.
  7. The installation checks for the existence of a logical volume at /dev/qalab/pristine-[release]-root.  If it exists, it creates a new snapshot at /dev/qalab/root and reboots. If it does not, continues with installation and goto step 4.
  8. System reboots, Juju installs and redeploys nova-compute to a fresh Ubuntu installation.

This process takes place on all nodes in parallel.  With it in place, we were able to cut down the time it took to tear-down and re-provision a node from ~30 minutes to 10 to 15 minutes depending on the service being deployed.

By taking this approach we are also minimize the chance of any nodes hitting an archive inconsistency during installation. This is a known issue when deploying the development release and halts installation on any node that hits it, failing the entire deployment.

All of this is embedded in debian-installer preseeds via Cobbler snippets.  The snippets and kick starts are available at lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot.

In the future, we’ll be investigating the use of kexec as an alternative to reboot after snapshot restoration to reduce the time spent waiting on servers to boot.  This should minimize the test cycle even more. Credit to James Blair for the idea (see http://amo-probos.org/post/11).

Management of Jenkins

All of the projects in Jenkins are managed using Jinja2 XML templates in-conjunction with python-jenkins (python-jenkins); this makes it really easy to setup new jobs in the lab and reconfigure existing ones as required (as well as providing great backup!).

Templates and management scripts can be found in lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Testing Openstack Essex on Ubuntu Precise

This testing was the first to be setup in the lab.  Jenkins (using the git plugin) monitors the upstream github.com repositories for commits on the master branch.  When a change is detected the following process is triggered:

Build

Objective: Validate that upstream trunk still builds OK with current packaging for Ubuntu.

  1. A new snapshot upstream tarball is generated based on the latests commit to the upstream component.
  2. The latest archive packaging for the component is pulled in from lp:~ubuntu-server-dev/<COMPONENT>/essex
  3. Any changes in the testing packaging for the component are merged from lp:~openstack-ubuntu-testing/<COMPONENT>/essex
  4. New changelog entries are automatically created for the new upstream commits.
  5. The source package is generated and built in a clean schroot using sbuild locally.

On the assumption that the package built OK locally:

  1. The source package is uploaded to the Testing PPA (ppa:openstack-ubuntu-testing/testing)
  2. The testing packaging branch is push back to lp:~openstack-ubuntu-testing/<COMPONENT>/essex.
  3. The binary packages from the sbuild are installed into the local reprepro managed archive.

This process is managed by a single script (tarball.sh); Credit to Chuck Short for pulling together this part of the process based on work from Openstack upstream.

For changes to the nova project the deploy phase is then executed.

Deploy

Objective: Validate that packages install, can be configured and reach a know good state prior to execution of testing.

This phase of testing uses Juju with Cobbler to deploy Openstack into the QA lab infrastructure; It utilizes branches of the Openstack charms to support use of a local archive along with a deployer wrapper around Juju written by Adam Gandelman which executes the actual deployment using Juju and monitors for errors.

The deployer is configured to know where to get the right codebase for the Openstack charms, which services to deploy and which relations to setup between services. As you can see from the above diagram this is non-trivial but the charms and Juju do most of the hard work.

Once Openstack is deployed successfully the test phase is then executed.

Test

Objective: Validate that the Openstack deployment in the lab actually works!

At this point, we can run any integration tests we wish against the newly deployed cloud.  This testing is able to help us achieve multiple goals:

  • Early detection of upstream bugs that break Openstack functionality on Ubuntu
  • Verification that packaging branches in the development version of Ubuntu are compatible with upstream trunk.
  • Using these packages, verification that our Juju charms are deploying a functional Openstack cloud and are up-to-date with any deployment-related configuration changes upstream.

At the moment this phase looks like this:

  1. Configure the Openstack deployment (Adams deployer script provides some utility functions for locating specific services in the environment)
    • Creates network configuration in Nova for the private instance network as well as a pool of public floating IPs.
    • Upload an image into the Glance server for use during testing
    • Creates EC2 credentials in the Keystone server for use during testing.
  2. Run the devstack exercise test scripts which ensure basic functionality of the deployment. Currently, this includes:
    • Basic euca-tools EC2 API for starting and stopping instances
    • EC2 AMI bundle uploads
    • Floating IP allocation, association and connectivity to instance
    • Volume creation and attachment to instance

Note: These are the same sets of tests that are currently run against proposed commits to gerrit upstream.

Longer term we aim to use the Openstack Tempest test suite in the lab; Adam is currently working on getting this up and running.

Reporting

The Jenkins instance in the QA lab is not publicly accessible; however all jobs run in the lab are published out (using the Jenkins build-publisher plugin) to http://jenkins.qa.ubuntu.com so that people can see the current state of the testing packaging in Ubuntu precise.

We are also working on setting up email notifications.

Success so far

Juju charms deploy Openstack components in a configuration that is compatible with upstream trunk prior to updates to packaging in Ubuntu.  Previously packages were updated in the archive first while Juju charm updates lagged behind as incompatibilities were uncovered after the fact.

We enabled automated testing 2 days prior to the 3rd Essex milestone release.  We were able to uncover and help fix a handful of bugs upstream before the release, including critical bugs like 921784.  In the past, these bugs were typical uncovered after the release (both upstream and in Ubuntu).

Since E3, there have been even more critical bugs uncovered by this testing and fixed upstream, some of which are only applicable to Ubuntu-specific configurations (not tested upstream) and would have been uncovered by users after code hit the Ubuntu archive (See 922232).

Further Plans for the Lab

Pre-commit  testing of changes to stable branches;  The Ubuntu Server team are  working upstream on maintaining the stable branches of released versions  of OpenStack – this work will validate patches proposed to stable  branches in review.openstack.org against the current version of the  packaging in released versions of Ubuntu.  Initially this will target  Diablo on Ubuntu 11.10 but will also support Essex on Ubuntu 12.04 once  released.  Ideally the testing process will provide feedback on  review.openstack.org to help the stable release team review proposed  patches.

References

Jenkins job configurations: lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Scripts supporting the lab: lp:~openstack-ubuntu-testing/+junk/jenkins-scripts

LVM snapshot preseeds and Cobbler snippets: lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot

All other relevant scripts, charm branches, etc: https://code.launchpad.net/~openstack-ubuntu-testing/

Credits

Overall management of delivery and general whip cracking: Dave Walker

Lab installation and base configuration: Pete Graner, Tim Gardner, Brad Figg, James Page

Fence agent for network power control of servers: Andres Rodriguez

Source package creation and build process: Chuck Short and James Page

Deployment testing using Juju: Adam Gandelman

Testing of Openstack: Adam Gandelman

Jenkins packaging, configuration and management: James Page

Gerrit Plugin for pre-commit testing and generally great ideas: Monty Taylor and James Blair

Writing and reviewing this post: Adam Gandelman, Chuck Short and Dave Walker.

Comment » | Uncategorized

Developer Weekly Activity Review – August 5, 2011

August 5th, 2011 — 7:22pm

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activities

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

This is select list of topics discussed this week in the developer mailing list and is not a complete list.  Please visit the archive to see all the topics.

  • Dashboard newbie question – Joshua Harlow asks about getting the Dashboard to work with the latest version of Nova; he is getting a 400 https response when deploying from the dashboard. Devin Carlen recommended using the Diablo-3 milestone release. Thierry Carrez asked that any issues be added to the bug report, which Carlo Impagliazzo listed for, but 820972. This issue is still being discussed at https://lists.launchpad.net/openstack/msg03430.html.
  • [Gerrit] Getting “invalid author” - Yuriy Taraday indicated that he is trying to use Gerrit to propose changes to Keystone. He is following the instructions from the wiki and is getting “invalid author” errors. James Blair indicated that the issue was due to how Launchpad was importing account information and made a change, which solved the initial problem seen by Yuriy. Yuriy then indicated that he is seeing a “missing Change-ID in commit message” error. Several suggestions were presented as solutions. More details on those options at https://lists.launchpad.net/openstack/msg03421.html.
  • Questions About test_localization.py – Zed Shaw indicated that he is going to clean up tests and source files and has questions about the tests/test_localization.py file and what its purpose is. The discussion on  the purpose of the test and some ways to beef up what is tested were discussed.

Statistics

For the latest on development activities on OpenStack please check these sites for more details:

1 comment » | Communication, Development, Newsletter

OpenStack Developer Activity Review (February 21 – February 28)

February 28th, 2011 — 10:17am

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activities
• Nova 2011.1.1 release candidate available for testing - http://nova.openstack.org/tarballs/nova-2011.1.1~bzr653.tar.gz

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

Announcing the OpenStack Dashboard – Devin Carlen announced the availability of the OpenStack Dashboard based on the NASA Nebula Dashboard. Code links at https://lists.launchpad.net/openstack/msg01069.html.
Availability of RHEL build of Bexar release of OpenStack Nova – Grid Dynamics announced the availability of OpenStack Nova RHEL 6.0 build. The instructions to install on run on their build at http://wiki.openstack.org/NovaInstall/RHEL6Notes.
Working on fixing code after a review? Please mark merge proposal Work in Progress – Jay Pipes commented on the backlog of code waiting for review at https://code.launchpad.net/nova/+activereviews. Jay suggests several steps for developers whose code has been reviewed with suggested code fixes. Ewan Mellor suggested that a new Wiki page be created with all the information for developers on “How Peer Reviews Work”.
Should the OpenStack API re-use the EC2 credentials? – Justin Santa Barbara submitted a new blueprint, https://blueprints.launchpad.net/nova/+spec/authentication-consistency, to handle the authentication issues that arose between how OpenStack API and EC2 worked. More than 20 responses on this topic were posted so see https://lists.launchpad.net/openstack/msg00917.html for more information.
Decoupling of Network and Compute Services for the new Network Service design – Ryu Ishimoto proposes a possible solution to make NOVA no longer directly dependent on the network services to allow for future “pluggable” options. More than 20 responses on this topic as well – see https://lists.launchpad.net/openstack/msg00908.html for more information.

Statistics

  • Number of OpenStack Developers on Contributors List – 151
  • Cactus Release Status – Blueprints (http://wiki.openstack.org/releasestatus/)
    • Essential – 5 Design Approved; 1 Implemented – 1 Needs Code Review – 1 Beta Available
    • High – 11 Blueprints; 2 Drafting
    • Medium – 22 Blueprints; 3 Needs Code Review; 1 Blocked
    • Low – 16 Blueprints; 2 Needs Code Review; 4 Implemented

For the latest on development activities on OpenStack please check these sites for more details:

1 comment » | Development, Newsletter

OpenStack Developer Activity Review (January 29 – February 4)

February 4th, 2011 — 1:37pm

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activities

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

Statistics

  • Number of OpenStack Developers on Contributors List – 139 (+2 for week)
  • Cactus Release Status – 25 Blueprints (http://wiki.openstack.org/releasestatus/)
    • 4 Essential; 2 High; 4 Medium; 7 Low; 8 Undefined

For the latest on development activities on OpenStack please check these sites for more details:

1 comment » | Communication, Development

OpenStack Developer Activity Review (January 21 – 28)

January 28th, 2011 — 2:21pm

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activities

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

  • State of Glance? -  Jay Pipes gives an overview of all features developed for Bexar in Project Glance as well as the current thinking for the Cactus Project Glance release. More details at https://lists.launchpad.net/openstack/msg00358.html
  • Network Service for L2/L3 Network Infrastructure Blueprint – Ewan Mellor responded to interest from several developers on the status of the bexar-network-service blueprint and its status. The blueprint will be postponed to Cactus and a discussion about this blueprint was started.
  • Cactus Release Preparation – Rick Clark gives an overview of the strategy behind the Cactus release and requests that all new features for Cactus be proposed to the community by February 3, 2011. The Cactus release schedule is http://wiki.openstack.org/CactusReleaseSchedule

Statistics

For the latest on development activities on OpenStack please check these sites for more details:

1 comment » | Communication, Development, Newsletter

OpenStack Developer Activity Review (Jan 14 – 21)

January 21st, 2011 — 1:37pm

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activities

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

  • Can ec2-api-tools maintain OpenStack Nova? -  Kenji Funasaki asked if it is possible to get image information by using ec2-describe-images functions within the EC2-API-TOOLS? Soren Hansen and Masanori ITOH responded that ec2 api tools use the SOAP based API for EC2 which are in development for the Cactus release in April.
  • Windows/Hyper-V – Jordan Rinke inquired about Hyper-V running on NOVA and was looking for more information; Chiradeep Vittal supplied a wiki page http://wiki.openstack.org/HypervInstall.
  • New Config File for Nova-API – Armando Migliaccio asked about the new way of configuring nova-api as the old glags style has been replaced by past.deploy style.  Jay Pipes, Andy Smith, and Anne Gentle responded on the need to improve documentation in this area; Armando did respond that flags and paste.deploy don’t configure the same things and he had confusion from the way he as using –flagfile=nova-api.conf.
  • Live Migration Branch Status – Thierry Carrez posted information on the decision to delay the release of this code from Bexar to Cactus. Masumotok confirmed this decision.
  • Deprecating nova-objectstore – Thierry Carrez posted information on the long term plan to remove the objectstore post Bexar and leverage Glance. Further discussion on the technical method to achieve this is discussed in the email thread…

Statistics

For the latest on development activities on OpenStack please check these sites for more details:

2 comments » | Communication, Newsletter

OpenStack Developer Activity Review

January 14th, 2011 — 9:38am

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activities

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

  • Feature Freeze Branches Needing Review – Thierry presented a list of branches needing review in priority order by project at https://lists.launchpad.net/openstack/msg00269.html.  Nova had 14 branches for review; Glance had 1 branch; Swift had 1 branch. A discussion about the process to review branches by developers across projects is planned to find a better way to reduce the high number of un-reviewed branches in Nova for future releases.
  • Lazy Import of Modules – Ewan Mellor opened an issue about creating a standard methodology for loading modules on demand to eliminate unnecessary dependencies for un-needed modules. Responses from Jay Pipes, Vishvananda Ishaya, Devin Carlen, and Andy Smith discussed several solutions but it appears that the idea from Andy Smith will move forward at this time (see https://lists.launchpad.net/openstack/msg00281.html)
  • Nova Translators – Jay Pipes announced that Monty Taylor and himself have completed the support for i18n in Nova and language translation has begun for NOVA into Austrian, Chinese (Simplified), Danish, Italian, Japanese, Russian, and Ukranian. If you are interested in assisting with another language please go to https://translations.launchpad.net/nova/trunk/+pots/nova.
    • Other Languages underway include Brazilian Portuguese

Statistics

For the latest on development activities on OpenStack please check these sites for more details:

3 comments » | Communication, Development

OpenStack Developer Activity Weekly Review

January 7th, 2011 — 12:12pm

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activitites

  • Branch Merge Proposal Freeze (January 6, 2011) – Only impacts branches introducing new features or changes in expected behavior; not branches fixing bugs. To submit a new feature you must follow the exception process found at http://wiki.openstack.org/BranchMergeProposalFreeze
  • Feature Freeze for Bexar scheduled for January 13th

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

  • BZR Plugin – Josh Kearney has created a plugin (https://github.com/jk0/bzr-pre-commit-hook) to ensure that PEP8 and unit tests are run automatically when submitting code
  • OpenStack Programming Model Framework – John Purrier has proposed (https://lists.launchpad.net/openstack/msg00225.html)  a model where developers will  interact with the OpenStack services will not interact directly with the service API’s, but rather will have a set of published language bindings that define the programming model. This does not preclude direct service calls, but this will be discouraged in favor of using the bindings. The bindings will be considered the “Nova API” for all intents and purposes. Further discussion on this issue led to the community not being in favor of the idea of only using language bindings for developers as there is no guarantee that a specific language binding would be available for a specific release.
  • OpenStack API Discussion – John Purrier has posted (https://lists.launchpad.net/openstack/msg00181.html) some comments on the OpenStack API discussing availability for Bexar and/0r Cactus release which also prompted the community focus on what the internal APIs are for OpenStack developers and what the external APIs are for  ecosystem developers to leverage. The current naming is devAPI for internal OpenStack development and OpenStack API for people creating solutions around OpenStack. Please note, naming of the internal devAPI is subject to change. Further details on OpenStack API for Bexar can be found at https://blueprints.launchpad.net/nova/+spec/openstack-api-parity.

For the latest on development activities on OpenStack please check these sites for more details:

Comment » | Development

Back to top