Event Details


Evaluation of hardware-specific machine learning systems orchestrated in an OS cluster at NIST

ML systems are composed of complex combinations of algorithmic solutions applied to process large amounts of data, which requires time and computation resources. Implementations of such systems often rely on various external dependencies. Thereby evaluating such systems may end up being a real challenge.

Containers are efficient means to package such systems but do not carry with them hardware passthroughs for programs configured for a specific hardware; Virtual Machines (VM) offer a better way to guarantee this replicability.

In our work, an entrypoint has been designed to allow the execution of the VMs regardless of their implementation. The interfaced VMs have then been coupled with a batch processing orchestrator, that schedules their execution in an OpenStack project.

As a result, we are able to schedule the execution of any pair of system/dataset over a cluster and therefore optimize the allocation of resources. This setup has been successfully used during a NIST evaluation.


What can I expect to learn?

The Multimodal Information Group, which is part of the National Institute of Standards and Technology, often performs advanced evaluations and benchmarking of the performances of computer programs.

Evaluations are performed on complex systems that often rely on various external dependencies. These systems may have to be executed multiple times on various data sets,which requires a lot of computation resources. Also, in this case, programs have been configured on a specific node, with a specific set of hardware components.

The planning of such evaluations raise a lot of different questions, the following ones will be covered through a use case:

  • How to standardize the distribution format of the systems to guarantee their replicability and reusability?
  • How to design an entrypoint to allow multiple executions of the same system?
  • How to efficiently schedule these different executions on a cluster?

This session may be useful for anyone who would like to know about the distribution of hardware-specific machine learning systems, as well as automating batch processing on an Openstack cluster.

Tuesday, April 30, 2:15pm-2:25pm
Difficulty Level: Intermediate
Guest Researcher @NIST
Software engineer at the National Institute of Standards and Technology since 2017, originally from France. Science and tech enthusiat, always looking for new knowledge. FULL PROFILE
Guest researcher @NIST
Software engineer at the National Institute of Standards and Technology since 2018. Love design patterns. Tech and Karate enthusiast. FULL PROFILE
National Institute of Standards and Technology
Jim Golden is a IT Specialist at the National Institute of Standards and Technology (NIST) and has been a cloud operator since 2013. Jim operates heterogenious GPU enabled clouds which run some of NIST’s scientific evaluations, which include deep neural network training in speech recognition and detection tasks in multi-camera streaming video. FULL PROFILE