What's all about Azure Event Grid?

Azure Event Grid is one of the newest products available in Azure cloud stack. Since it's still in preview, we are not offered full functionality(so e.g. only two regions can be selected, not all event publishers have been added). However with all the goodnes provided by this component, we can start thinking about "reactive programming in the cloud" - at least this is what documentation tells us. Let's dive deeper into Event Grid and find why it's so special.

Competitors

Event Grid is all about events. You may ask how it is different comparing similar products like Event Hub or Service Bus. If you take a look at the basic architecture, you'll find very similar concept like topics or subscribers (well at least for Service Bus). So why do I need Event Grid(which will complicate my architecture even more) when I can easily connect e.g. my Azure Functions to a topic and achieve the same functionality with ease? Well, this is only partially true.

Event Grid functional model(source: https://docs.microsoft.com/en-us/azure/event-grid/overview)

The downside of other solutions is the need of pooling - details doesn't matter now, you have to implement some way of communication between your app and an event publisher. It can be long-pooling, event sourcing, WebSockets - whatever works can be used. So even if you establish a persistent connection, you have to talk to the other side and await messages. You're not passive in this model - that's why you cannot "react" on events passed to you. You only parse them and pass further.

Event Grid allows you to make your components "passive" - they are somewhere in the cloud and are only interested in the data you send to them. They don't have to persist any connections - it's up to Event Grid to distribute messages and deliver to the configured subscribers. Microsoft states, that this approach is suited for serverless scenario and I can agree with them - you can make underlying infrastructure even more abstract and control the flow of event from the single point. For me the possibility to configure connection between Event Hub and several Azure Functions using Azure Portal(so I don't have to pass a connection string of EH to each individual component) is definitely a big YES to Event Grid.

Should I go for it?

I still think, that though Event Grid simplifies and improves working with serverless architecture(what am I saying - it actually enables you to start thinking about serverless at all...), you cannot just take it, write a couple of Functions and say "this is how we're making applications today in our company". It still requires proper planning, it's still not valid for each and every application(with Event Hub, Event Grid and Azure Functions, you may assume, that an event will reach its destination... at some point in time) and forces you to change your mindset into being "reactive"(and this is sometimes a challenge itself).

Event Grid as the "man-in-the-middle" in serverless architecture(source: https://docs.microsoft.com/en-us/azure/event-grid/overview)

On the other hand I like how it smoothly integrates with the cloud - for now only a few publishers are available, but we're given a promise, that this will change soon. I treat it as a serverless orchestrator - it's the centre of my architecture, where I can separate concerns seamlessly. Combine it with negligible cost($0.60 per million operations, first 100k is free) and easy learning curve and ask yourself why haven't you tested it yet?

Achieving consistency in Azure Table Storage #2

In the previous post I presented some easy ways of achieving consistency in Table Storage by storing data within one partition. This works in the majority of scenarios, there're some cases however, when you have to divide records(because of the design, to preserve scalability or to logically separate different concerns) and still ensure, that you can perform operations within a transaction. You may be a bit concerned - all in all we just talked, that storing data within a single(at least from the transaction point of view) partition is required to actually be able to perform EGTs. Well - as always there's a solution to go beyond some limits and achieve what we're aiming for. Let's go!

Eventually consistent transactions

Just a note before we start - this pattern won't guarantee, that a transaction is isolated. This means that a client will be able to read data while a transaction is being processed. Unfortunately there's no easy way to completely lock tables while an inter-partition operation is being performed.

Let's back to our eventual consistency. What does it mean? The answer is pretty simple - once a transaction is finished, our data can be considered consistent. All right - but this is something new. What's the difference between transaction performed as EGT? 

In EGT your are performing maximally 100 operations without a possibility to see an ongoing process. In other words - you always see the result of a transaction. With eventual consistency you can divide the process into steps:

  • get an entity from Partition 1
  • inserty an entity into Partition 2
  • delete an entity from Partition 1

Of course you can have more than only 3 steps. The crux here is the clear division between each step. If we consider other operations performed during a transaction:

  • get an entity from Partition 1
  • get an entity from Partition 2
  • inserty an entity into Partition 2
  • get an entity from Partition 1
  • delete an entity from Partition 1

The whole view should be clearer. With eventual consistency those bolded steps stand for operations, which clearly are victims of read phenomenas. Always consider possible drawbacks of solutions like this and if needed, use other database which isolates transactions.

Achieving eventual consistency

To achieve eventual consistency we have to introduce a few other components to our architecture. We'll need at least two things:

  • queue which holds actions, which should be performed in a transaction
  • worker roles, which reads messages from a queue and perform the actual transactions

Now let's talk about each new component in details.

Queue

By using a queue we're able to easily orchestrate operations, which should be performed by worker roles. The easiest example is creating a project, which will archive records stored in Table Storage. Thanks to a queue we can post a message saying 'Archive a record', which can be read by other components and processed. Finally workers can post their messages saying, that an action has been finished. 

Worker role

When we're saying about workers we think about simple services, which perform some part of a flow. In eventual consistency pattern they're responsible for handling a transaction logic. If we come back to the example from the previous point, a worker would be responsible for moving an entity from the one table to another and then deleting it. The important note here is idempotence - you have to ensure, that you won't add more than one instance of an entity in the case of restarting the flow. The same goes when deleting things - you should delete only if an entity exists.

Considerations

You can apply this pattern not only to perform operations between different partitions - it also works when you're working with other components like blobs. It has some obvious drawbacks like lack of isolation or external segment, which have to be handled in your code. On the other hand it's a perfectly valid approach, especially in table-other_storage scenario.