Fan out/fan in Pattern - Part 3 - Azure Durable Functions

If you need to define a workflow that scales, by having multiple functions running in parallel the Fan out/fan in Pattern is the application pattern to use with Durable Functions.

This is a series where I am explaining the components of durable functions¹ and the different application patterns of implementation.

Check the PlayGoKids repository for this article demo.

Understanding the Flow

The Fan out/fan in pattern allows the processing of activities in parallel, which means multiple functions are spun to execute the activities, and at the end of processing the orchestrator waits for all parallel activities to finish. In this example, we get multiple Orders for Processing, and at the end, we send a notification. They are represented by the following activities: GetOrders, ProcessOrder and SendNotification.

The FanOutIn_HttpStart is the HTTP Trigger.

Starter Function (aka DurableClient)

[FunctionName($"{nameof(FanOutIn)}_HttpStart")]
public async Task<HttpResponseMessage> HttpStart(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestMessage req,
    [DurableClient] IDurableOrchestrationClient starter,
    ILogger log)
{
    var instanceId = await starter.StartNewAsync(nameof(Orchestrator));

    log.LogInformation($"Started orchestration with ID = '{instanceId}'.");

    return starter.CreateCheckStatusResponse(req, instanceId);
}

This is an HTTPTrigger function that can be triggered from this CURL request:

curl --location --request GET 'http://localhost:7071/FanOutIn_HttpStart'

Orchestrator Function

The Orchestrator controls all the activities. It loads Orders from GetOrders and then it processes the Orders by calling multiple ProcessOrder activities in a foreach loop. In the end, it retrieves all orderNumbers and calls the activity SendNotification to finalize the orchestrator. Check the PlayGoKids repository for details of IOrderService.

[FunctionName(nameof(Orchestrator))]
public async Task RunOrchestrator(
    [OrchestrationTrigger] IDurableOrchestrationContext context)
{
    var orderProcessingTasks = new List<Task<string>>();

    // Get a list of orders
    var orders = await context.CallActivityAsync<List<Order>>(nameof(IOrderService.GetOrders), DateTime.Now);

    foreach (var order in orders)
    {
        // Process orders in parallel
        Task<string> task = context.CallActivityAsync<string>(nameof(IOrderService.ProcessOrder), order);
        orderProcessingTasks.Add(task);
    }

    // Wait for all orders to process
    var orderNumbers = await Task.WhenAll(orderProcessingTasks);

    // Send notification
    await context.CallActivityAsync(nameof(IOrderService.SendNotification), orderNumbers);
}

Activity Functions

The activities were created in order Top-down, displayed below:

public class Activity
{
    private readonly IOrderService _orderService;

    public Activity(IOrderService orderService)
    {
        _orderService = orderService;
    }

    [FunctionName(nameof(IOrderService.GetOrders))]
    public List<Order> GetOrders([ActivityTrigger] DateTime dateTime, ILogger log)
    {
        log.LogInformation($"Getting orders.");
        var orders = _orderService.GetOrders(dateTime);
        return orders;
    }

    [FunctionName(nameof(IOrderService.ProcessOrder))]
    public string ProcessOrder([ActivityTrigger] Order order, ILogger log)
    {
        log.LogInformation($"Processing order {order.OrderNumber}");
        var orderNumber = _orderService.ProcessOrder(order);
        return orderNumber;
    }

    [FunctionName(nameof(IOrderService.SendNotification))]
    public void SendNotification([ActivityTrigger] string[] orderNumbers, ILogger log)
    {
        log.LogInformation($"Send notification.");
        _orderService.SendNotification(orderNumbers);
    }
}

The OrderService is just a dummy that implements the Activity actions. It doesn't do much:

public class OrderService : IOrderService
{
    public List<Order> GetOrders(DateTime dateTime)
    {
        //TODO: Retrieve orders for a particular date

        return new List<Order>()
        {
            new Order(){OrderNumber = "ON-00001"},
            new Order(){OrderNumber = "ON-00002"},
            new Order(){OrderNumber = "ON-00003"},
            new Order(){OrderNumber = "ON-00004"},
            new Order(){OrderNumber = "ON-00005"}
        };
    }

    public string ProcessOrder(Order order)
    {
        //TODO: Any processing
        return order.OrderNumber;
    }

    public void SendNotification(string[] orderNumbers)
    {
        //TODO: Send notification
    }
}

This is an example of how to use the Fan out/fan in pattern, applied to the processing of Orders.

In the next article of the series let's check out how to use the async HTTP API pattern.

Full Series

Major components

Function chaining

Fan out/fan in

Async HTTP APIs

Monitor

Human interaction

Aggregator (stateful entities)