This workshop paper is joined work with Hewlett-Packard Laboratories (Palo Alto, CA) and will be presented at the 8th International Workshop on Self-Managing Database Systems (SMDB 2013) in conjunction with
29th IEEE International Conference on Data Engineering (ICDE) in Brisbane (Australia) from 8th-12th April 2013.
The performance of intra-node parallel dataflow programs in the context of streaming systems depends mainly on two parameters: the degree of parallelism for each node of the dataflow program as well as the batching size for each node. In the state-of-the-art systems the user has to specify those values manually. Manual tuning of both parameters is necessary in order to get good performance. However, this process is difficult and time consuming---even for experts. In this paper we introduce and optimization algorithm that optimizes both parameters automatically. We define a novel cost model for intra-node parallel dataflow programs with user-defined functions. Furthermore, we introduce different batching schemes to reduce the number of output buffers, i.e., main memory consumption. We implemented our approach on top of the open source system Storm and ran experiments with different workloads. Our results show a throughput improvement of more than one order of magnitude while the optimization time is less than a second.