Starting fun with Durable Functions

Not so long ago I started developing a simple side project related to game development. Since it's just a proof-of-concept, I've decided, that I'll present here sneak peeks with the work done. There're two main parts of this project:

  • cloud backend based on serverless architecture
  • desktop client built with Electron and HTML5

As I said it's more like a concept than a real product so all bleeding edge technologies will be used just to confirm that "it's possible" :)

Starting with Durable Functions

Orchestrating workflows in Azure Functions is pretty new feature, which is still in the preview stage. The good thing is the fact, that it's currently somehow documented(not very good but still better than nothing) and draws bigger and bigger audience to FaaS built with Azure components. You can find all required details regarding installation and concepts of this functionality here. We'll go straight to the implementation.

Consider following diagram:

If you're familiar with Durable Task Framework, those names should be familiar in some way. We're telling the framework to start an orchestration, which will schedule activities in the proper order and perform some work. Note that communication between Client and Orchestration is not bidirectional. The idea here is to orchestrate work and don't wait for it. What we're given however is a link to the orchestration itself, where we can check the current status and result. 

The important thing here is that Orchestration aggregates data from Activity 1 and Activity 2 before going further. This allows for easy decoupling of responsibilities, so there's no problem when it comes to ordering tasks.

Let's check the real world example.

Implementation

I converted the above diagram to the corresponding functions:

/
public static class Create
{
	[FunctionName("Galaxy_Create_Start")]
	public static async Task<HttpResponseMessage> StartOrchestration(
		[HttpTrigger(AuthorizationLevel.Function, methods: "post", Route = "orchestrators/run")] HttpRequestMessage req,
		[OrchestrationClient] DurableOrchestrationClient starter,
		TraceWriter log)
	{
		// Function input comes from the request content.
		dynamic eventData = await req.Content.ReadAsAsync<object>();
		string instanceId = await starter.StartNewAsync("Galaxy_Create", eventData);

		log.Info($"Started orchestration with ID = '{instanceId}'.");

		return starter.CreateCheckStatusResponse(req, instanceId);
	}

	[FunctionName("Galaxy_Create")]
	public static async Task<string> RunImpl([OrchestrationTrigger] DurableOrchestrationContext context)
	{
		var result = await Task.WhenAll(context.CallActivityAsync<string>("Utility_Coords"),
			context.CallActivityAsync<string>("Utility_Name"));

		await context.CallActivityAsync("Galaxy_Create_Impl", new CreateGalaxyContext(result[1], result[0]));

		return "Galaxy created!";
	}

	[FunctionName("Galaxy_Create_Impl")]
	public static async Task CreateGalaxy(
		[ActivityTrigger] CreateGalaxyContext context,
		[Table("galaxies")] IAsyncCollector<GalaxyDataEntity> galaxies)
	{
			await galaxies.AddAsync(new GalaxyDataEntity(context.Name, context.Coords));                
	}
}

If I start an orchestration, I'll get a new galaxy added to my table in Table Storage like so:

Let's analyze each function one by one.

Galaxy_Create_Start

It's our Orchestration Client. It acts as an interface to the orchestration, se we can start it. As you can see it works as a HttpTrigger - you can use whichever trigger you like like Event Hub or Event Grid. It returns a simple payload, which allow you to interact with the working orchestration:

{
    "id": "8d5c8fdf0cf54d87839d9d17223268bf",
    "statusQueryGetUri": "http://localhost:7071http://localhos/admin/extensions/DurableTaskExtension/instances/8d5c8fdf0cf54d87839d9d17223268bf?taskHub=SampleHubVS&connection=Storage&code=YgC1NtQXBBRJGO9sGbNWVA03nVjedvjPwPUWZ/lH3vIBVcngQIDYjg==",
    "sendEventPostUri": "http://localhost:7071/admin/extensions/DurableTaskExtension/instances/8d5c8fdf0cf54d87839d9d17223268bf/raiseEvent/{eventName}?taskHub=SampleHubVS&connection=Storage&code=YgC1NtQXBBRJGO9sGbNWVA03nVjedvjPwPUWZ/lH3vIBVcngQIDYjg==",
    "terminatePostUri": "http://localhost:7071/admin/extensions/DurableTaskExtension/instances/8d5c8fdf0cf54d87839d9d17223268bf/terminate?reason={text}&taskHub=SampleHubVS&connection=Storage&code=YgC1NtQXBBRJGO9sGbNWVA03nVjedvjPwPUWZ/lH3vIBVcngQIDYjg=="
}

Galaxy_Create

The actual orchestration. It holds logic responsible for scheduling all tasks as needed.

Galaxy_Create_Impl

The actual implementation of a worker, which is responsible for inserting data into a storage table. It's unaware of other functions - it just expects a particular context passed to it, which hold the necessary data.

There're other 2 functions in this solution(Utility_Coords and Utility_Name), but they're just a standard ActivityTriggers, which generate random data. 

Conclusion

As you can see Durable Functions ease developing organized flows greatly. Without this feature we'd have to use e.g. queues or perform HTTP requests to gather all the data. What is more it moves a solution towards event driven architectures and simplifies overall structure. In the next post we'll convert this solution to a passive application with Event Grid taking the role of the commander. Stay tuned!

Durable Functions - Durable Task Framework basics

In the previous post I presented some basic concepts behind Durable Functions and reasoning why they've been introduced and what we can achieve with them. This time I'll focus on the very foundation of Durable Functions - Durable Task Framework - and its features. We'll try to understand its mechanics and build a very simple workflow to get an idea how it works. Let's go!

Mechanics

The basic concept of Durable Task Framework is to use Service Bus to orchestrate work and use it as a temporary storage for a state. When Framework is initialized, it creates a Service Bus queue under the hood, which will be used as a the main components to pass messages. Note that queue size is not unrestricted - you have to choose on of the following values:

  • 1024MB
  • 2048MB
  • 3072MB
  • 4096MB
  • 5120MB

any other size will be treated as an error and will result in an exception.

 If you go to Core Concepts sections for Durable Task Framework, you can find following diagram:

https://github.com/Azure/durabletask/wiki/images/concepts.png

It shows the underlying structure of Durable Task Framework and all main elements of the architecture. It may be a little confusing now, but once we start creating orchestrations, all will become easier to understand.

The most important thing here is Task Hub Worker, which allows adding Task Orchiestrations and Task Activities and dispatching to these - to make the long story short, it acts as foundation of your solution.

The difference between Orchiestration and Activity is fairly simple - once Activity is the actual action, which should be performed and we can refer to it as a simple and atomic task, which will be executed, Orchiestration is thing, which aggregates Activities and orchestrates them. You can think about it as a conductor, which is responsible for going the right path.

From Durable Tasks to Durable Functions

You may ask "how do Durable Tasks connect to Durable Functions?" - in fact initially there's no explicit connection. We have to consider what would be the best way to achieve orchestration in the world of serverless. In the previous post I mentioned, that current solution includes using Azure Storage queues, what for sure lets you achieve the goal, but is far from ideal solution. Natural evolution of this idea is to utilize something what is called event sourcing and instead of pushing and fetching messages from queues, just raise an event and wait for an eventual response:

  • Function1 started executing
  • Function1 called Function2
  • Function2 started executing
  • Function2 finished executing
  • Function1 called Function3
  • Function1 finished executing

This is a trivial concept but yet a really powerful one. By storing a state in a such manner(using an append-only log) you're gaining many profits:

  • there's no way to mutate a state with appending another event
  • immutable state - difficult to corrupt
  • no locking
  • it' easy to recreate a state if needed 

Now if you consider, that Activities can be treated as events, there's an easy way to Durable Functions, where each Activity is another function and a state is stored in Azure Storage and maintained by the runtime.

Summary

Today we went a bit deeper into Durable Task Framework and considered connection between this library and Durable Functions. In the next post I'll try to present a basic example of Durable Function and what changes in that approach when creating serverless application.