The OpenStack Blog

Category: Development

Project Glance Webinar Recording Available

Over 57 community enthusiasts joined Jay Pipes, Project Glance Team Leader, and James Weir, CTO of UShareSoft, for a 1 hour overview the project as well as a discussion on the future direction for Glance and cloud computing. A recorded session of the webinar is available at http://cc.readytalk.com/play?id=d7aki1. For more general information on Glance, visit http://www.openstack.org/projects/image-service/.


.

Clustered LVM on DRBD resource in Fedora Linux

(Crossposted from Mirantis Official Blog)

As Florian Haas has pointed out in my previous post’s comment, our shared storage configuration requires special precautions to avoid corruption of data when two hosts connected via DRBD try to manage LVM volumes simultaneously. Generally, these precautions concern locking LVM metadata operations while running DRBD in ‘dual-primary’ mode.

Let’s examine it in detail. The LVM locking mechanism is configured in the [global] section of /etc/lvm/lvm.conf. The ‘locking_type’ parameter is the most important here. It defines which locking LVM is used while changing metadata. It can be equal to:

  • ’0′: disables locking completely – it’s dangerous to use;
  • ’1′: default, local file-based locking. It knows nothing about the cluster and possible conflicting metadata changes;
  • ’2′: uses an external shared library and is defined by the ‘locking_library’ parameter;
  • ’3′: uses built-in LVM clustered locking;
  • ’4′: read-only locking which forbids any changes of metadata.

The simplest way is to use local locking on one of the drbd peers and to disable metadata operations on another one. This has a serious drawback though: we won’t have our Volume Groups and Logical Volumes activated automatically upon creation on the other, ‘passive’ peer. The thing is that it’s not good for the production environment and cannot be automated easily.

But there is another, more sophisticated way. We can use the Linux-HA (Heartbeat) coupled with the LVM Resource Agent. It automates activation of the newly created LVM resources on the shared storage, but still provides no locking mechanism suitable for a ‘dual-primary’ DRBD operation.

It should be noted that full support of clustered locking for the LVM can be achieved by the lvm2-cluster Fedora RPM package stored in the repository. It contains the clvmd service which runs on all hosts in the cluster and controls LVM locking on shared storage. In this case, we have only 2 drbd-peers in the cluster.

clvmd requires a cluster engine in order to function properly. It’s provided by the cman service, installed as a dependency of the lvm2-cluster (other dependencies may vary from installation to installation):

(drbd-node1)# yum install clvmd
...
Dependencies Resolved

================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
lvm2-cluster x86_64 2.02.84-1.fc15 fedora 331 k
Installing for dependencies:
clusterlib x86_64 3.1.1-1.fc15 fedora 70 k
cman x86_64 3.1.1-1.fc15 fedora 364 k
fence-agents x86_64 3.1.4-1.fc15 updates 182 k
fence-virt x86_64 0.2.1-4.fc15 fedora 33 k
ipmitool x86_64 1.8.11-6.fc15 fedora 273 k
lm_sensors-libs x86_64 3.3.0-2.fc15 fedora 36 k
modcluster x86_64 0.18.7-1.fc15 fedora 187 k
net-snmp-libs x86_64 1:5.6.1-7.fc15 fedora 1.6 M
net-snmp-utils x86_64 1:5.6.1-7.fc15 fedora 180 k
oddjob x86_64 0.31-2.fc15 fedora 61 k
openais x86_64 1.1.4-2.fc15 fedora 190 k
openaislib x86_64 1.1.4-2.fc15 fedora 88 k
perl-Net-Telnet noarch 3.03-12.fc15 fedora 55 k
pexpect noarch 2.3-6.fc15 fedora 141 k
python-suds noarch 0.3.9-3.fc15 fedora 195 k
ricci x86_64 0.18.7-1.fc15 fedora 584 k
sg3_utils x86_64 1.29-3.fc15 fedora 465 k
sg3_utils-libs x86_64 1.29-3.fc15 fedora 54 k

 

Transaction Summary
================================================================================
Install 19 Package(s)

The only thing we need the cluster for is the use of clvmd; the configuration of cluster itself is pretty basic. Since we don’t need advanced features like automated fencing yet, we specify manual handling. As we have only 2 nodes in the cluster, we can tell cman about it. Configuration for cman resides in the /etc/cluster/cluster.conf file:

<?xml version="1.0"?\>
<cluster name="cluster" config_version="1"\>
  <!-- post_join_delay: number of seconds the daemon will wait before
        fencing any victims after a node joins the domain
  post_fail_delay: number of seconds the daemon will wait before
        fencing any victims after a domain member fails
  clean_start    : prevent any startup fencing the daemon might do.
        It indicates that the daemon should assume all nodes
        are in a clean state to start. --\>
  <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
  <clusternodes>
   <clusternode name="drbd-node1" votes="1" nodeid="1">
    <fence>
    <!-- Handle fencing manually -->
     <method name="human">
      <device name="human" nodename="drbd-node1"/>
     </method>
    </fence>
   </clusternode>
   <clusternode name="drbd-node2" votes="1" nodeid="2">
    <fence>
    <!-- Handle fencing manually -->
     <method name="human">
      <device name="human" nodename="drbd-node2"/>
     </method>
    </fence>
   </clusternode>
  </clusternodes>
  <!-- cman two nodes specification -->
  <cman expected_votes="1" two_node="1"/>
  <fencedevices>
  <!-- Define manual fencing -->
   <fencedevice name="human" agent="fence_manual"/>
  </fencedevices>
</cluster>

clusternode name should be a fully qualified domain name and should be resolved by DNS or be present in /etc/hosts. Number of votes is used to determine quorum of the cluster. In this case, we have two nodes, one vote per node, and expect one vote to make the cluster run (to have a quorum), as configured by cman expected attribute.

The second thing we need to configure is the cluster engine (corosync). Its configuration goes to /etc/corosync/corosync.conf:

compatibility: whitetank
totem {
  version: 2
  secauth: off
  threads: 0
  interface {
    ringnumber: 0
    bindnetaddr: 10.0.0.0
    mcastaddr: 226.94.1.1
    mcastport: 5405
  }
}
logging {
  fileline: off
  to_stderr: no
  to_logfile: yes
  to_syslog: yes
  # the pathname of the log file
  logfile: /var/log/cluster/corosync.log
  debug: off
  timestamp: on
logger_subsys {
  subsys: AMF
  debug: off
}
}
amf {
  mode: disabled
}

The bindinetaddr parameter must contain a network address. We configure corosync to work on eth1 interfaces, connecting our nodes back-to-back on 1Gbps network. Also, we should configure iptables to accept multicast traffic on both hosts.

It’s noteworthy that these configurations should be identical on both cluster nodes.

After the cluster has been prepared, we can change the LVM locking type in /etc/lvm/lvm.conf on both drbd-connected nodes:

global {
  ...
  locking_type = 3
  ...
}

Start cman and clvmd services on drbd-peers and get our cluster ready for the action:

(drbd-node1)# service cman start
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Unfencing self... [ OK ]
Joining fence domain... [ OK ]
(drbd-node1)# service clvmd start
Starting clvmd:
Activating VG(s): 2 logical volume(s) in volume group "vg-sys" now active
2 logical volume(s) in volume group "vg_shared" now active
[ OK ]

Now, as we already have a Volume Group on the shared storage, we can easily make it cluster-aware:

(drbd-node1)# vgchange -c y vg_shared

Now we see the ‘c’ flag in VG Attributes:

(drbd-node1)# vgs
VG        #PV #LV #SN Attr    VSize   VFree
vg_shared   1   3   0 wz--nc  1.29t   1.04t
vg_sys      1   2   0 wz--n-  19.97g  5.97g

As a result, Logical Volumes created in the vg_shared volume group will be active on both nodes, and clustered locking is enabled for operations with volumes in this group. LVM commands can be issued on both hosts and clvmd takes care of possible concurrent metadata changes.

OpenStack Nova: basic disaster recovery

We have published a new blog post about handling basic issues and virtual machines recovery methods. From the blog:

Today, I want to take a look at some possible issues that may be encountered while using OpenStack. The purpose of this topic is to share our experience dealing with the hardware or software failures which definitely would be faced by anyone who attempts to run OpenStack in production.

Read the complete blog at http://mirantis.blogspot.com/2011/06/openstack-nova-basic-disaster-recovery.html.

OpenStack Glance Webinar

Jay Pipes, Project Glance PTL and James Weir, CTO of UShareSoft.com are hosting “Future Direction and Discussion on Glance”; the webinar is scheduled for June 21, 2011 at Noon EST. To register for this webinar, please sign up at https://cc.readytalk.com/cc/schedule/display.do?udc=i5l6gkl36wsy.

This webinar will provide an introduction to the project followed by an open community discussion on the roadmap for the Glance project.

Developer Activity Review – May 20

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activities

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

This is select list of topics discussed this week in the developer mailing list and is not a complete list.  Please visit the archive to see all the topics.

  • Graphical Console for VMs w/o Network Stacks – Donal Lafferty is looking for a blueprint for a graphical console instead of text console. Mike Scherbakov replied with instructions for using the current Dashboard.
  • Unassigned Essential Diablo Specs – Thierry Carrez listed a set of blueprints that are planned for Diablo but currently have no owner. The list is at https://lists.launchpad.net/openstack/msg02432.html.
  • Python-novaclient vs. python-openstack.compute - Soren Hansen brings up the issue that we have 2 distinct but similar libraries doing the same thing; Sandy Walsh, Dan Prince, Gabe Westmaas, Ed Leafe, and Josh Kearney all joined in the discussion of how to handle this issue.
  • Global deployment of Glance – Glen Campbell raises the issue of having replicas of Glance in multiple installations and how to maintain synch. Jay Pipes, Chris Behrens, and Eric Day discussed the idea of using SWIFT to manage the multiple zones issue and offload the problem from Glance.

Statistics

For the latest on development activities on OpenStack please check these sites for more details:

Mirantis: OpenStack Deployment on Fedora using Kickstart

The team at Mirantis published a new blog post today on deploying OpenStack on Fedora (http://mirantis.blogspot.com/2011/05/openstack-deployment-on-fedora-using.html). From the blog:

In this article, we discuss our approach to performing an Openstack installation on Fedora using our RPM repository and Kickstart. When we first started working with OpenStack, we found that the most popular platform for deploying OpenStack was Ubuntu, which seemed like a viable option for us, as there are packages for it available, as well as plenty of documentation. However, our internal infrastructure is running on Fedora, so instead of migrating the full infrastructure to Ubuntu, we decided to make OpenStack Fedora-friendly. The challenge in using Fedora, however, is that there aren’t any packages, nor is there much documentation available. Details of how we worked around these limitations are discussed below.

The complete blog post with detailed step by step directions at http://mirantis.blogspot.com/2011/05/openstack-deployment-on-fedora-using.html.

Announcing Project RedDwarf – Database as a Service

From Daniel Morris in the OpenStack developer mailing list comes a new incubation project announcement…

Today Rackspace is announcing the introduction of Database as a Service (Project RedDwarf) for a possible affiliated OpenStack incubation project.

To give you some background, Database as a Service is a scalable relational database service that allows users toquickly and easily utilize the features of a relational database without the burden of handling complex administrative tasks.   With this service, cloud users and database administrators can provision and manage multiple database instances as needed.

Initially, our plan is for the service to focus on providing resource isolation at high performance while automating complex administrative tasks including deployment, configuration, patching, backups, restores, and monitoring. Some of the key features for the first release are listed below:

  • Single tenant MySQL instance with unlimited databasesper instance
  • Public API’s to create, read, update, and delete databases and database users
  • User and database access management with root user access
  • Scale database instance memory sizes up and down
  • Scale up storage sizes
  • Database backups and restores
  • Instance migrations
  • Instance metrics and monitoring

This represents our current thinking for a first release, and we welcome feedback from the community on any suggested features.

While still in the early stages of development, the service is already tightly integrated with OpenStack Compute (Nova).  We chose Nova because the component-based architecture and open standards make it the perfect virtualization layer for our product platform.  The compute layer provides the reusable and deployable services needed to build an extensible service deployment foundation that will be used to deliver not only a MySQL database service, but also many other services in the future.

The initial architecture of this service is being designed around several technologies listed below

  • Open Stack Compute (Nova)
  • OpenVZ – OpenVZ is a container based virtualization technology that ensures guaranteed resource minimums and maximums delivering exceptional performance from a MySQL server, comparable to a bare metal box.
  • Guest Agent – The guest agent is the management interface to the container (VM). All operations originating from the Cloud Databases API use the guest to manipulate the container.

More details can be found on the blueprint and wiki, please take a look, get involved, and provide any comments or feedback you may have.  Also, join us during our session at the OpenStack Design Summit in Santa Clara April 26th – 29th to learn more about our approach, ask questions and become active in the project!  We are excited about working with the community as we continue to develop relational databases in the cloud.

OpenStack Developer Activity Weekly Review (April 9- 15)

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activities

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

This is select list of topics discussed this week in the developer mailing list and is not a complete list.  Please visit the archive to see all the topics.

Statistics

  • Number of OpenStack Developers on Contributors List – 175 (+10 for week)
  • Final Cactus Release Status – Blueprints (http://wiki.openstack.org/releasestatus/)
    • Essential – 5 Design Approved; 5 Implemented
    • High – 12 Blueprints; 9 Implemented – 3 Deferred
    • Medium – 20 Blueprints; 17 Implemented – 3 Deferred
    • Low – 15 Blueprints; 8 Implemented – 7 Deferred

For the latest on development activities on OpenStack please check these sites for more details:

Community Weekly Newsletter (April 9 – 15)

OpenStack Community Newsletter – April 15, 2011

This weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please email stephen.spector@openstack.org.

Todd Willey – On Break from Documenting OpenStack

HIGHLIGHTS

EVENTS

DEVELOPER COMMUNITY

GENERAL COMMUNITY

COMMUNITY STATISTICS (4/8– 4/15)

  • Data Tracking Graphs – http://wiki.openstack.org/WeeklyNewsletter
  • OpenStack Compute (NOVA) Data
    • 45 Active Reviews
    • 242 Active Branches – owned by 60 people & 13 teams
    • 1,947 commits by 69 people in last month
  • OpenStack Object Storage (SWIFT) Data
    • 11 Active Reviews
    • 62 Active Branches – owned by 23 people & 5 teams
    • 174 commits by 12 people in last month
  • Twitter Stats for Week:  #openstack 156 total tweets; OpenStack 448 total tweets  (does not include RT)
  • Bugs Stats for Week: 343 Tracked Bugs; 66 New Bugs; 50 In-process Bugs; 1 Critical Bugs; 11 High Importance Bugs; 215 Bugs (Fix Committed)
  • Blueprints Stats for Week:  195 Blueprints; 3 Essential, 11 High, 12 Medium, 20 Low, 149 Undefined
  • OpenStack Website Stats for Week:  12,164 Visits, 26,959 Pageviews, 57.43% New Visits
    • Top 5 Pages: Home 43.85%; /projects 10.84%; /projects/compute 16.11%; /projects/storage 11.36%; /Community 7.21%

OPENSTACK IN THE NEWS

OpenStack Announces Cactus Release

With the availability of the Cactus release of OpenStack today the momentum and progress of the project continues to grow. A tremendous amount of effort and contribution from the large, and growing, community has added significant features, fixed a lot of bugs, and debated and discussed many technical issues. I am impressed with the progress that has been made since the Bexar release just 10 weeks ago and believe the projects and code are tracking to fill the promise of being the ubiquitous, open source cloud solution.

New features in Nova (OpenStack Compute) include:

  • Two additional virtualization technologies: LXC containers and VMWare/vSphere ESX / ESXi 4.1, Update 1. Driven by a common compute control infrastructure (Nova) this brings the options for OpenStack host virtualization to 8 (adding to Microsoft Hyper-V, KVM, QEMU, UML, Xen, and Citrix XenServer).
  • Live Migration support for KVM-based systems landed in the Cactus release; it is now possible to move running VMs from one physical host to another without a shut down.
  • Lots of new features were added to XenServer support: network and file injection, IPv6 support, instance resize and rescue, network QoS, and VM instance parameters.
  • The OpenStack Compute API version 1.0 is available, with the OpenStack Compute API version 1.1 marked as “experimental” for Cactus. The intent is to finalize the 1.1 API at the Diablo design summit and have it complete and stable in the Diablo release. Multi-tenant accounting support was added to OpenStack API, allowing multiple accounts (projects) and admin API access to create accounts & users.
  • The OpenStack Compute API version 1.1 supports a standardized extension mechanism, this allow developers to innovate more quickly by adding extensions to their local OpenStack installations ahead of the code being accepted by the OpenStack community as a whole;
  • Nova can now start instances from VHD images that include customer data and kernel in one unified image.
  • Volume backend support has been enhanced; Nova now supports volumes residing on HP SANs and Solaris iSCSI devices.
  • Continued work on feature uniformity and parity across network types and hypervisors; IPv6 is now supported in all network modes, including FlatManager and VlanNetworkManager. Basic network injection is now supported under XenAPI.
  • Multi-cluster region support, which allows administrators to manage servers in clusters, and create fault zones and availability zones.

New features in Glance (OpenStack Image Registry and Delivery) include:

  • New command line interface tool (aptly-named “glance”) that allows direct access to Glance services through the API.
  • Support for multiple image formats through a new disk_format and container_format metadata definition.
  • Uploaded images can now be verified against a client-provided checksum, to ensure the integrity of the transfer.

New features in Swift (OpenStack Object Storage) include:

  • The option to serve static website content directly from a Swift installation using container listings in index.html displays. Swift will automatically translate requests to possible /index.html resolutions, where the index.html display is configurable per container.
  • To more quickly detect errors for often-served files, Swift now performs content checksum validation during object GET actions.
  • Performance of many request types has been improved through a refactoring of the Swift Proxy Server.
  • To avoid slowdowns for common operations when deleted items build up over time, Swift now has improved indexing of the SQLite databases for account and container listing and tracking.
  • An enhanced authentication system (SWauth) is available.
  • The ability to collect and serve data that enables integration of service provider billing solutions or internal chargebacks.

In addition to the work done on the project code, there have been several other things happening to improve the state of OpenStack. Primary amongst these was the election of Project Team Leaders for the three current OpenStack projects… Congratulations to Vish Ishaya (vishy) [Nova], John Dickinson (notmyname) [Swift], and Jay Pipes (jaypipes) [Glance] as new PTL’s, they also join the OpenStack Project Policy Board.

The OpenStack Project Policy Board also had elections, with 5 board members holding elected seats. These are Thierry Carrez (ttx), Rick Clark (dendrobates), Eric Day (eday), Soren Hansen (soren), and Ewan Mellor (ewanmellor). Congratulations folks!

OpenStack has defined a process for bringing in new projects, both as core projects and those that are being incubated.  (See http://wiki.openstack.org/Governance/Approved/NewProjectProcess). The initial incubation project is “Burrow”, a simple queuing service for OpenStack being led by Eric Day (eday). At the upcoming Diablo Design Summit I expect several more projects to be proposed for incubation; including Load Balancing and Database Services.

The Diablo Design Summit is setting up to be the most dynamic and content-filled summit to date! The entire week is completely filled with attendees and items for discussion. While ttx and the PTL’s are busy scheduling all the sessions here are a few of the highlights:

  • Network as a Service. In order to fulfill the vision of OpenStack as a secure cloud infrastructure with the ability to federate across clouds it is imperative that the underlying network support isolation, federation, and the ability to manage these topologies. The NaaS discussion has many important participants working hard to collaborate on this very technical set of issues.
  • Volume services. Extending the initial Nova volume management for richer block storage solutions.
  • Additional machine types (GPU accelerators, larger multi-core processor systems).
  • Consistent authentication and authorization across OpenStack projects.
  • Multi-zone support, intra-data center and federation across data centers.
  • Project management discussions.
  • Stability and QA automation. A key theme of the Diablo release will be to automate the build and test infrastructure for OpenStack to ensure that trunk is always runnable. With the proliferation of virtualization architectures, machine architectures, and service options this will be a key element to success of the project.
  • Complete and stable OpenStack version 1.1 API.
  • Target large scale service provider deployments, with proof of concepts happening in large OpenStack contributor sites.

A job well done to all of the folks that contributed and made the Cactus release come together and get released. I will see all of you at the Design Summit in Santa Clara and look forward to the discussions around the Diablo release and the future of OpenStack!

Lastly, the OpenStack Project Team Leaders are hosting a Webinar on Tuesday April 19th at 3:00 pm CST. More information at http://www.openstack.org/blog/2011/04/openstack-cactus-webinar/.

John
Director, OpenStack@Rackspace

Back to top