The OpenStack Blog

Category: Development

Developer Activity Review (June 17 – 24)

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activities

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

This is select list of topics discussed this week in the developer mailing list and is not a complete list.  Please visit the archive to see all the topics.

  • Feedback on Portable Configuration Drive Blueprint – Thorsten von Eicken provided feedback to the Configuration Drive blueprint at https://blueprints.launchpad.net/nova/+spec/configuration-drive.  Scott Moser, Paul Voccio (blueprint author), and Vish Ishaya provided feedback and discussion
  • Swift+Keystone error – Tres Henry reported issues in trying to get a Swift+Keystone development environment setup. Jesse Andrews responded with an updated Keystone GitHub link. Todd Wiley responded with a proxy-server.conf setting to try; account_autocreate = true.  Tres responded with the error showing up even after trying this configuration file setting.

Statistics

For the latest on development activities on OpenStack please check these sites for more details:

Project Glance Webinar Recording Available

Over 57 community enthusiasts joined Jay Pipes, Project Glance Team Leader, and James Weir, CTO of UShareSoft, for a 1 hour overview the project as well as a discussion on the future direction for Glance and cloud computing. A recorded session of the webinar is available at http://cc.readytalk.com/play?id=d7aki1. For more general information on Glance, visit http://www.openstack.org/projects/image-service/.


.

Clustered LVM on DRBD resource in Fedora Linux

(Crossposted from Mirantis Official Blog)

As Florian Haas has pointed out in my previous post’s comment, our shared storage configuration requires special precautions to avoid corruption of data when two hosts connected via DRBD try to manage LVM volumes simultaneously. Generally, these precautions concern locking LVM metadata operations while running DRBD in ‘dual-primary’ mode.

Let’s examine it in detail. The LVM locking mechanism is configured in the [global] section of /etc/lvm/lvm.conf. The ‘locking_type’ parameter is the most important here. It defines which locking LVM is used while changing metadata. It can be equal to:

  • ’0′: disables locking completely – it’s dangerous to use;
  • ’1′: default, local file-based locking. It knows nothing about the cluster and possible conflicting metadata changes;
  • ’2′: uses an external shared library and is defined by the ‘locking_library’ parameter;
  • ’3′: uses built-in LVM clustered locking;
  • ’4′: read-only locking which forbids any changes of metadata.

The simplest way is to use local locking on one of the drbd peers and to disable metadata operations on another one. This has a serious drawback though: we won’t have our Volume Groups and Logical Volumes activated automatically upon creation on the other, ‘passive’ peer. The thing is that it’s not good for the production environment and cannot be automated easily.

But there is another, more sophisticated way. We can use the Linux-HA (Heartbeat) coupled with the LVM Resource Agent. It automates activation of the newly created LVM resources on the shared storage, but still provides no locking mechanism suitable for a ‘dual-primary’ DRBD operation.

It should be noted that full support of clustered locking for the LVM can be achieved by the lvm2-cluster Fedora RPM package stored in the repository. It contains the clvmd service which runs on all hosts in the cluster and controls LVM locking on shared storage. In this case, we have only 2 drbd-peers in the cluster.

clvmd requires a cluster engine in order to function properly. It’s provided by the cman service, installed as a dependency of the lvm2-cluster (other dependencies may vary from installation to installation):

(drbd-node1)# yum install clvmd
...
Dependencies Resolved

================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
lvm2-cluster x86_64 2.02.84-1.fc15 fedora 331 k
Installing for dependencies:
clusterlib x86_64 3.1.1-1.fc15 fedora 70 k
cman x86_64 3.1.1-1.fc15 fedora 364 k
fence-agents x86_64 3.1.4-1.fc15 updates 182 k
fence-virt x86_64 0.2.1-4.fc15 fedora 33 k
ipmitool x86_64 1.8.11-6.fc15 fedora 273 k
lm_sensors-libs x86_64 3.3.0-2.fc15 fedora 36 k
modcluster x86_64 0.18.7-1.fc15 fedora 187 k
net-snmp-libs x86_64 1:5.6.1-7.fc15 fedora 1.6 M
net-snmp-utils x86_64 1:5.6.1-7.fc15 fedora 180 k
oddjob x86_64 0.31-2.fc15 fedora 61 k
openais x86_64 1.1.4-2.fc15 fedora 190 k
openaislib x86_64 1.1.4-2.fc15 fedora 88 k
perl-Net-Telnet noarch 3.03-12.fc15 fedora 55 k
pexpect noarch 2.3-6.fc15 fedora 141 k
python-suds noarch 0.3.9-3.fc15 fedora 195 k
ricci x86_64 0.18.7-1.fc15 fedora 584 k
sg3_utils x86_64 1.29-3.fc15 fedora 465 k
sg3_utils-libs x86_64 1.29-3.fc15 fedora 54 k

 

Transaction Summary
================================================================================
Install 19 Package(s)

The only thing we need the cluster for is the use of clvmd; the configuration of cluster itself is pretty basic. Since we don’t need advanced features like automated fencing yet, we specify manual handling. As we have only 2 nodes in the cluster, we can tell cman about it. Configuration for cman resides in the /etc/cluster/cluster.conf file:

<?xml version="1.0"?\>
<cluster name="cluster" config_version="1"\>
  <!-- post_join_delay: number of seconds the daemon will wait before
        fencing any victims after a node joins the domain
  post_fail_delay: number of seconds the daemon will wait before
        fencing any victims after a domain member fails
  clean_start    : prevent any startup fencing the daemon might do.
        It indicates that the daemon should assume all nodes
        are in a clean state to start. --\>
  <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
  <clusternodes>
   <clusternode name="drbd-node1" votes="1" nodeid="1">
    <fence>
    <!-- Handle fencing manually -->
     <method name="human">
      <device name="human" nodename="drbd-node1"/>
     </method>
    </fence>
   </clusternode>
   <clusternode name="drbd-node2" votes="1" nodeid="2">
    <fence>
    <!-- Handle fencing manually -->
     <method name="human">
      <device name="human" nodename="drbd-node2"/>
     </method>
    </fence>
   </clusternode>
  </clusternodes>
  <!-- cman two nodes specification -->
  <cman expected_votes="1" two_node="1"/>
  <fencedevices>
  <!-- Define manual fencing -->
   <fencedevice name="human" agent="fence_manual"/>
  </fencedevices>
</cluster>

clusternode name should be a fully qualified domain name and should be resolved by DNS or be present in /etc/hosts. Number of votes is used to determine quorum of the cluster. In this case, we have two nodes, one vote per node, and expect one vote to make the cluster run (to have a quorum), as configured by cman expected attribute.

The second thing we need to configure is the cluster engine (corosync). Its configuration goes to /etc/corosync/corosync.conf:

compatibility: whitetank
totem {
  version: 2
  secauth: off
  threads: 0
  interface {
    ringnumber: 0
    bindnetaddr: 10.0.0.0
    mcastaddr: 226.94.1.1
    mcastport: 5405
  }
}
logging {
  fileline: off
  to_stderr: no
  to_logfile: yes
  to_syslog: yes
  # the pathname of the log file
  logfile: /var/log/cluster/corosync.log
  debug: off
  timestamp: on
logger_subsys {
  subsys: AMF
  debug: off
}
}
amf {
  mode: disabled
}

The bindinetaddr parameter must contain a network address. We configure corosync to work on eth1 interfaces, connecting our nodes back-to-back on 1Gbps network. Also, we should configure iptables to accept multicast traffic on both hosts.

It’s noteworthy that these configurations should be identical on both cluster nodes.

After the cluster has been prepared, we can change the LVM locking type in /etc/lvm/lvm.conf on both drbd-connected nodes:

global {
  ...
  locking_type = 3
  ...
}

Start cman and clvmd services on drbd-peers and get our cluster ready for the action:

(drbd-node1)# service cman start
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Unfencing self... [ OK ]
Joining fence domain... [ OK ]
(drbd-node1)# service clvmd start
Starting clvmd:
Activating VG(s): 2 logical volume(s) in volume group "vg-sys" now active
2 logical volume(s) in volume group "vg_shared" now active
[ OK ]

Now, as we already have a Volume Group on the shared storage, we can easily make it cluster-aware:

(drbd-node1)# vgchange -c y vg_shared

Now we see the ‘c’ flag in VG Attributes:

(drbd-node1)# vgs
VG        #PV #LV #SN Attr    VSize   VFree
vg_shared   1   3   0 wz--nc  1.29t   1.04t
vg_sys      1   2   0 wz--n-  19.97g  5.97g

As a result, Logical Volumes created in the vg_shared volume group will be active on both nodes, and clustered locking is enabled for operations with volumes in this group. LVM commands can be issued on both hosts and clvmd takes care of possible concurrent metadata changes.

OpenStack Nova: basic disaster recovery

We have published a new blog post about handling basic issues and virtual machines recovery methods. From the blog:

Today, I want to take a look at some possible issues that may be encountered while using OpenStack. The purpose of this topic is to share our experience dealing with the hardware or software failures which definitely would be faced by anyone who attempts to run OpenStack in production.

Read the complete blog at http://mirantis.blogspot.com/2011/06/openstack-nova-basic-disaster-recovery.html.

OpenStack Glance Webinar

Jay Pipes, Project Glance PTL and James Weir, CTO of UShareSoft.com are hosting “Future Direction and Discussion on Glance”; the webinar is scheduled for June 21, 2011 at Noon EST. To register for this webinar, please sign up at https://cc.readytalk.com/cc/schedule/display.do?udc=i5l6gkl36wsy.

This webinar will provide an introduction to the project followed by an open community discussion on the roadmap for the Glance project.

Developer Activity Review – May 20

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activities

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

This is select list of topics discussed this week in the developer mailing list and is not a complete list.  Please visit the archive to see all the topics.

  • Graphical Console for VMs w/o Network Stacks – Donal Lafferty is looking for a blueprint for a graphical console instead of text console. Mike Scherbakov replied with instructions for using the current Dashboard.
  • Unassigned Essential Diablo Specs – Thierry Carrez listed a set of blueprints that are planned for Diablo but currently have no owner. The list is at https://lists.launchpad.net/openstack/msg02432.html.
  • Python-novaclient vs. python-openstack.compute - Soren Hansen brings up the issue that we have 2 distinct but similar libraries doing the same thing; Sandy Walsh, Dan Prince, Gabe Westmaas, Ed Leafe, and Josh Kearney all joined in the discussion of how to handle this issue.
  • Global deployment of Glance – Glen Campbell raises the issue of having replicas of Glance in multiple installations and how to maintain synch. Jay Pipes, Chris Behrens, and Eric Day discussed the idea of using SWIFT to manage the multiple zones issue and offload the problem from Glance.

Statistics

For the latest on development activities on OpenStack please check these sites for more details:

Mirantis: OpenStack Deployment on Fedora using Kickstart

The team at Mirantis published a new blog post today on deploying OpenStack on Fedora (http://mirantis.blogspot.com/2011/05/openstack-deployment-on-fedora-using.html). From the blog:

In this article, we discuss our approach to performing an Openstack installation on Fedora using our RPM repository and Kickstart. When we first started working with OpenStack, we found that the most popular platform for deploying OpenStack was Ubuntu, which seemed like a viable option for us, as there are packages for it available, as well as plenty of documentation. However, our internal infrastructure is running on Fedora, so instead of migrating the full infrastructure to Ubuntu, we decided to make OpenStack Fedora-friendly. The challenge in using Fedora, however, is that there aren’t any packages, nor is there much documentation available. Details of how we worked around these limitations are discussed below.

The complete blog post with detailed step by step directions at http://mirantis.blogspot.com/2011/05/openstack-deployment-on-fedora-using.html.

Announcing Project RedDwarf – Database as a Service

From Daniel Morris in the OpenStack developer mailing list comes a new incubation project announcement…

Today Rackspace is announcing the introduction of Database as a Service (Project RedDwarf) for a possible affiliated OpenStack incubation project.

To give you some background, Database as a Service is a scalable relational database service that allows users toquickly and easily utilize the features of a relational database without the burden of handling complex administrative tasks.   With this service, cloud users and database administrators can provision and manage multiple database instances as needed.

Initially, our plan is for the service to focus on providing resource isolation at high performance while automating complex administrative tasks including deployment, configuration, patching, backups, restores, and monitoring. Some of the key features for the first release are listed below:

  • Single tenant MySQL instance with unlimited databasesper instance
  • Public API’s to create, read, update, and delete databases and database users
  • User and database access management with root user access
  • Scale database instance memory sizes up and down
  • Scale up storage sizes
  • Database backups and restores
  • Instance migrations
  • Instance metrics and monitoring

This represents our current thinking for a first release, and we welcome feedback from the community on any suggested features.

While still in the early stages of development, the service is already tightly integrated with OpenStack Compute (Nova).  We chose Nova because the component-based architecture and open standards make it the perfect virtualization layer for our product platform.  The compute layer provides the reusable and deployable services needed to build an extensible service deployment foundation that will be used to deliver not only a MySQL database service, but also many other services in the future.

The initial architecture of this service is being designed around several technologies listed below

  • Open Stack Compute (Nova)
  • OpenVZ – OpenVZ is a container based virtualization technology that ensures guaranteed resource minimums and maximums delivering exceptional performance from a MySQL server, comparable to a bare metal box.
  • Guest Agent – The guest agent is the management interface to the container (VM). All operations originating from the Cloud Databases API use the guest to manipulate the container.

More details can be found on the blueprint and wiki, please take a look, get involved, and provide any comments or feedback you may have.  Also, join us during our session at the OpenStack Design Summit in Santa Clara April 26th – 29th to learn more about our approach, ask questions and become active in the project!  We are excited about working with the community as we continue to develop relational databases in the cloud.

OpenStack Developer Activity Weekly Review (April 9- 15)

Many people have asked for more insight into the developer activities for OpenStack as the large number of code changes and proposals make it difficult to monitor everything happening. In hopes of exposing more of the developer activities, I plan to post a weekly or biweekly blog post on the latest development activities. If you have any ideas for this blog post, please email me at stephen.spector@openstack.org. I am always ready to listen to the community for new ideas.

Activities

Developer Mailing List (archive: https://lists.launchpad.net/openstack/)

This is select list of topics discussed this week in the developer mailing list and is not a complete list.  Please visit the archive to see all the topics.

Statistics

  • Number of OpenStack Developers on Contributors List – 175 (+10 for week)
  • Final Cactus Release Status – Blueprints (http://wiki.openstack.org/releasestatus/)
    • Essential – 5 Design Approved; 5 Implemented
    • High – 12 Blueprints; 9 Implemented – 3 Deferred
    • Medium – 20 Blueprints; 17 Implemented – 3 Deferred
    • Low – 15 Blueprints; 8 Implemented – 7 Deferred

For the latest on development activities on OpenStack please check these sites for more details:

Community Weekly Newsletter (April 9 – 15)

OpenStack Community Newsletter – April 15, 2011

This weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please email stephen.spector@openstack.org.

Todd Willey – On Break from Documenting OpenStack

HIGHLIGHTS

EVENTS

DEVELOPER COMMUNITY

GENERAL COMMUNITY

COMMUNITY STATISTICS (4/8– 4/15)

  • Data Tracking Graphs – http://wiki.openstack.org/WeeklyNewsletter
  • OpenStack Compute (NOVA) Data
    • 45 Active Reviews
    • 242 Active Branches – owned by 60 people & 13 teams
    • 1,947 commits by 69 people in last month
  • OpenStack Object Storage (SWIFT) Data
    • 11 Active Reviews
    • 62 Active Branches – owned by 23 people & 5 teams
    • 174 commits by 12 people in last month
  • Twitter Stats for Week:  #openstack 156 total tweets; OpenStack 448 total tweets  (does not include RT)
  • Bugs Stats for Week: 343 Tracked Bugs; 66 New Bugs; 50 In-process Bugs; 1 Critical Bugs; 11 High Importance Bugs; 215 Bugs (Fix Committed)
  • Blueprints Stats for Week:  195 Blueprints; 3 Essential, 11 High, 12 Medium, 20 Low, 149 Undefined
  • OpenStack Website Stats for Week:  12,164 Visits, 26,959 Pageviews, 57.43% New Visits
    • Top 5 Pages: Home 43.85%; /projects 10.84%; /projects/compute 16.11%; /projects/storage 11.36%; /Community 7.21%

OPENSTACK IN THE NEWS

Back to top