Dat tricky Throughput Unit

This is a perfect test whether you understand the service or not. Pardon me if this was/is obvious f

This is a perfect test whether you understand the service or not. Pardon me if this was/is obvious for you - apparently for me it was not. What's more, some people still seem to be confused. And we're here to avoid confusions. Let's start!

Hit the limit

Event Hub's model(both from technical and pricing point of view) is one of the easiest to understand and really, really straightforward. Let's perform a quick calculation:

I need to perform 1K ops / sec - I need 1 TU
I need to perform 10K ops / sec - I need 10TUs

In the overall price we can exclude cost related to the number of events processed(since it's like 5% of the money spent for this service). Now there's one more thing worth mentioning - TUs come as a pair - 1TU gives you 1MB(or 1K events) ingress and 2MB(or 2K events) egress. The question is:

"How to kill Event Hub's egress?"

Let's focus a little and try to find a scenario. We cannot exceed easily 1MB of egress having max ingress of 1MB. Maybe it'd doable by loading lots of data into EH and then introducing a consumer, which will be able to fetch and process 2MB of events per second. Still, this doesn't allow us to exceed the maximum of 2MBs of egress. We're safe.

But what if you introduce another *consumer group*? Since there's no filtering in Event Hub, each consumer group gets the same amount of events(in other words - when you have N consumer groups, you will read the stream N times). Now in the following scenario:

1MB of ingress
Consumer1(Consumer Group A)
Consumer2(Consumer Group B)

You've just hit the limit of 1TU(since you have 2MB of egress). Now let's try to scale and extend this solution. Let's introduce another consumer group:

1MB of ingress
Consumer1(Consumer Group A)
Consumer2(Consumer Group B)
Consumer3(Consumer Group C)

Now 1TU is not sufficient. By scaling out our Event Hub to 2TUs we can handle up to 4MBs of egress. In the same moment we can handle 2MBs of ingress. So if some reason throttling was your friend and kept the load up to some limit, you can quickly face problems and need to scale out once more.

Be smarter

As you can see, mistakes can be made and relying on consumer groups to filter(or orchestrate) events is not a way to go. In such scenario it'd much better to post events directly to e.g. Event Grid or use topics from Service Bus, so we can easily route messages. You have to understand, that the main purpose of Event Hub is to act as a really big pipe for data, which can be easily digested - misusing it could give you serious headaches.

 

Add comment