[dbis logo]

[Institut fuer Informatik] [Leerraum] [Humboldt-Universitaet zu Berlin]
07.07.2014 15:31 Alter: 5 yrs

New Publication: "Building a Transparent Batching Layer for Storm" (Technical Report)

Von: Matthias J. Sax

This technical report is joined work with Malu Castellanos from HP Labs (Palo Alto, CA).


Storm is a distributed intra-node-parallel stream processing system built for very low latency processing. One major drawback of Storm is its relatively low throughput. In order to increase Storm's throughput, we designed a batching layer for Storm that is able to improve Storm's throughput significantly. In order to get a high user acceptance, we did not modify Storm but build the batching layer "on top" of it. The layer is transparent to the Storm system as well as to the user code, i.e., the user-defined functions. Thus, already developed Storm programs (so-called topologies) can benefit from our batching layer without modification. In this document, we describe the design of the batching layer and provide inside into some implementation details