Boston
May 8-11, 2017
Optimizing Swift To Gain High Performance For LOSF In Product Environment
Swift suffers from significant performance degradation when it has to handle lots of small files(0~100KB) because objects are stored as separated files in backend. At iQiyi, we have developed a new durability interface on Hummingbird which is a reimplementation of Swift with Golang. Benchmark shows that write performance of the new interface is about 3X faster under heavy concurrent writes without sacrificing the read performance. And there is no degradation when the number of small objects grows which we believe is of great importance.
In this talk, we will discuss the design and implementation of the new durability interface such as
- Pack multiple small objects into a single POSIX file
- Manage metadata and index effectively with RocksDB
- Locate the objects without modifying current ring mechanism
- Incremental object replicator based on RocksDB transaction log to achieve eventual consistency
- Background garbage collector to reclaim deleted space
What can I expect to learn?
- Why Swift performance degrades when it comes to lots of small files applications
- The on disk format of the new disk file interface, namely about how to pack small objects
- How to manage the objects' metadata and index data with RocksDB
- How to locate the physical position of an object with current Ring mechanism and index
- How to implement a new incremental object replicator based on RocksDB transaction log to achieve eventual consistency
- How to implement a background garbage collector to reclaim deleted spaces
Wednesday, May 10, 11:00am-11:40am (3:00pm - 3:40pm UTC)
Difficulty Level: Intermediate
Storage Architect
Jeff has been working in storage area for nearly 4 years. He worked on several storage projects like vNAS, vBlob at VMware after his graduation from Fudan University. Later he joined iQiyi which is the largest online video site in China as a storage architect. His main task at iQiyi is contributing to transfer the whole storage infrastructure from POSIX filesystem access to object storage acess... FULL PROFILE
Scientist
Jieke Wu is Scientist in Cloud Group of iQIYI, role that he took in September 2012.iQIYI is the largest online video site in China which is established to build professional video experience. He managed a team to provide Infrastructure as a Service for private cloud, which supports internal developers with cloud computing, storage, and NFV etc. He spent ten years working in Linux related... FULL PROFILE
Download App
