Today, with great excitement, we proudly introduce a revolutionary update to the ArkFlow stream processing engine: full support for Python processors! This is not just a simple feature iteration; it marks a solid step towards more intelligent, user-friendly, and powerful real-time data processing. We are keenly aware of Python's immense influence and its vast ecosystem in the fields of artificial intelligence and machine learning. Now, with ArkFlow, you can seamlessly integrate all of this into high-performance, real-time stream processing pipelines.
In the recently merged PR, ArkFlow supports Python processors, which allows us to call Python scripts directly in ArkFlow. Using the Pyarrow library allows users to write data processing logic more freely. At the same time, relying on Python's powerful AI ecosystem, ArkFlow will also be able to call any Python machine learning/deep learning library (TensorFlow, PyTorch, etc.) and large models.
We will have more detailed articles to introduce relevant use cases in the future.
Hello, I am the founder of this project and I am very happy that a friend has shared it.
ArkFlow is positioned as a lightweight distributed stream processing engine that integrates streaming batches. With the help of datafusion's huge ecosystem and ArkFlow's scalable capabilities, we hope to build a huge data processing ecosystem to help the community simplify the threshold for data processing, because we always believe that flowing data can generate greater value.
Why do you have to beat a native performance-oriented streaming engine written in C or C++?
Currently, most of the mainstream stream processing engines are written in Java. Sorry, I may not add qualifiers to make you misunderstandings.
Software does not have silver bullets, so does programming languages, and each has its own strengths. I also like to use go and Java to develop software.