Watermark sense for windows7/14/2023 ![]() So, if we are keeping a count of the number of customers in a time window, the count we are outputting always represents the “count so far” - future later data could arrive and revise this count. Instead of thinking of getting a single answer for our windowed computation we need to think of getting a stream of revisions representing “the result so far”. This table will get updated as new events arrive. That is, it is a table where the key is the aggregation ID and the window time. Rather than having a lot of special constructs for windowing, we can just say that a windowed computation is going to be a table. Now that we’ve understood the streaming representation of a table we can understand how to generalize the dataflow model for handling windowed computation. This means that to maintain the current count per region I need to subtract one from the count for the old region and add one to the count for the new region.Ĭlearly being able to compute both on pure streams (like clicks) as well as streams of revisions (like customer location) is important. ![]() On the other hand, an update event (e.g., “Alice moved to Europe”) is revising the location information for the customer. The reason is that the count of clicks is never revised down - clicks arrive but old clicks never go away…so we only ever add to my count of clicks. In other words, another representation for the evolution of a table over time is a stream of the updates to the table:īut note that the mechanics of this computation are quite different than computing, say, the count of clicks by each customer. I can take the stream of updates to accounts and use it to keep a running count of the current number of customers in each region. We wrote about it in the first blog on Kafka Streams API. The answer is yes, there is a stream representation for a table like this. One approach to doing this would be to just run this count once a day, but can I do it in a streaming fashion? Let’s say that I have a table of customer accounts and I want to compute the number of customers in each geographical region. But most streaming processing systems don’t really represent tables of data. The task at hand for streaming apps isn’t processing only pure events, but combining it with data coming from these data stores. Most organizations have at their core a set of entities maintained in mutable databases - this might hold their customer account information, their sales, their inventory, etc. However, much of the data in an organization is not in this form. When people think of streams of events they mostly think about immutable entities. ![]() By way of doing that, let me introduce the concept of Tables in Kafka Streams API. The third critique requires outlining the more general thing this is a special case of. The second critique is that if we get specific about what we are trading off it is mostly non-functional characteristics and don’t really belong in your code at all but are more like tuning knobs. If there was no better solution we’d accept that this was unavoidable complexity, but I think there is a better approach, which I’ll outline. There are at least 8 varieties of trigger and several types of watermark. All this watermark business is complex, no denying that. The first critique is fairly straight-forward.
0 Comments
Leave a Reply. |