Abstract
In many application scenarios of distributed stream processing, there might be partial order relations among the requests. However, existing stream processing systems can not directly handle partially ordered requests, while indirect mechanisms are usually strongly coupled with business logic, which lack flexibility and have limited performance. We propose Pork, a novel distributed stream processing system targeting at partially ordered requests. In the experiments, the new system has achieved a parallelism and request throughput larger than the traditional mechanism in the presented example, and the performance overhead due to parallelism is considerably small. Then the scalability characteristic of the new system is discussed. What’s more, the experiment results also show that the new system has a more flexible load balancing ability.
W. Wu—This research is partially supported by National Natural Science Foundation of China (No. 61379157), Program of Science and Technology of Guangdong (No. 2015A010103007), Program of Science and Technology of Guangzhou (No. 201510010068), and Guangdong Frontier and Key Technology Innovation Fund (No. 2015B010111001).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This is a transitive, irreflexive and asymmetric relation, i.e. a strict partial order.
- 2.
Requests that should be processed directly prior to the current request.
- 3.
Either in deps or resolve of requests.
References
Apache Software Foundation: Apache Samza (2013). http://samza.apache.org/
Apache Software Foundation: Apache Flink (2015). http://flink.apache.org/
Ji, Y., Zhou, H., Jerzak, Z., Nica, A., Hackenbroich, G., Fetzer, C.: Quality-driven continuous query execution over out-of-order data streams. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 889–894. ACM (2015)
Kahn, A.B.: Topological sorting of large networks. Commun. ACM 5(11), 558–562 (1962)
Kreps, J., Narkhede, N., Rao, J.: Kafka: a distributed messaging system for log processing. In: NetDB (2011)
Lamport, L.: Time, clocks, and the ordering of events in a distributed system. Commun. ACM 21(7), 558–565 (1978)
Li, J., Tufte, K., Shkapenyuk, V., Papadimos, V., Johnson, T., Maier, D.: Out-of-order processing: a new architecture for high-performance stream systems. Proc. VLDB Endowment 1(1), 274–288 (2008)
Meehan, J., Tatbul, N., Zdonik, S., Aslantas, C., Cetintemel, U., Du, J., Kraska, T., Madden, S., Maier, D., Pavlo, A., et al.: S-Store: streaming meets transaction processing. Proc. VLDB Endowment 8(13), 2134–2145 (2015)
Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J.M., Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J., et al.: Storm@ Twitter. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 147–156. ACM (2014)
Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., Stoica, I.: Discretized streams: fault-tolerant streaming computation at scale. In: Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pp. 423–438. ACM (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Cai, R., Wu, W., Huang, N., Wu, L. (2016). Processing Partially Ordered Requests in Distributed Stream Processing Systems. In: Carretero, J., Garcia-Blas, J., Ko, R., Mueller, P., Nakano, K. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2016. Lecture Notes in Computer Science(), vol 10048. Springer, Cham. https://doi.org/10.1007/978-3-319-49583-5_16
Download citation
DOI: https://doi.org/10.1007/978-3-319-49583-5_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-49582-8
Online ISBN: 978-3-319-49583-5
eBook Packages: Computer ScienceComputer Science (R0)