What is stream-processing?
The processing of data in motion, or in other words, computing on data as it arrives.
Stream processing is a data processing technology used to query and process a continuous stream of data in real-time. Unlike batch processing, which waits for data to accumulate, stream processing acts on data packets as soon as they are generated by sources like sensors, IoT devices, or user activity logs.
graph LR
Center["What is stream-processing?"]:::main
Rel_throughput["throughput"]:::related -.-> Center
click Rel_throughput "/terms/throughput"
Rel_caching["caching"]:::related -.-> Center
click Rel_caching "/terms/caching"
Rel_etl_extract_transform_load["etl-extract-transform-load"]:::related -.-> Center
click Rel_etl_extract_transform_load "/terms/etl-extract-transform-load"
classDef main fill:#7c3aed,stroke:#8b5cf6,stroke-width:2px,color:white,font-weight:bold,rx:5,ry:5;
classDef pre fill:#0f172a,stroke:#3b82f6,color:#94a3b8,rx:5,ry:5;
classDef child fill:#0f172a,stroke:#10b981,color:#94a3b8,rx:5,ry:5;
classDef related fill:#0f172a,stroke:#8b5cf6,stroke-dasharray: 5 5,color:#94a3b8,rx:5,ry:5;
linkStyle default stroke:#4b5563,stroke-width:2px;
🧠 Teste de conhecimento
🧒 Explique como se eu tivesse 5 anos
Stream processing is like watching a movie as it streams over the internet. You don't wait for the whole movie to download (that would be batching). Instead, you watch each second of the movie as it arrives. You see the story happening 'live'.
🤓 Expert Deep Dive
Stream processing frameworks (like Apache Kafka, Flink, and Kinesis) handle 'unbounded' data. Key concepts include 'Windowing' (grouping data over a time period), 'Event Time' (when it happened) vs 'Processing Time' (when it reached the system), and handling out-of-order events.