Resilient Modern Apps with Polly in .NET - Retry Pattern

Modern Apps should be resilient, catering for transient faults when communicating with services. In this post, the retry pattern¹ is implemented with the help of Polly Retry and Polly Await Retry policies.

For more details about Polly Retry, please check their documentation². In all examples below, the Microsoft.Extensions.Http.Polly NuGet package version 7.0.10 was used.

Polly Retry

Sometimes a brute force approach to get the information does the job, and for that, Polly provides a Retry policy that allows for multiple contiguous retries.

The UML sequence diagram above shows the case of HTTP 429 (Too Many Requests) responses.

The code example below is extracted from an Azure Function Consumer project that is using a Typed HttpClient to communicate with an external service. The Polly Retry policy is applied to the HttpClient, and it will retry 3 times if the response is not successful.

Program.cs

using consumer.TypedHttpClient;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Polly;

var host = new HostBuilder()
    .ConfigureFunctionsWorkerDefaults()
    .ConfigureAppConfiguration((hostingContext, config) =>
    {
        var currentDirectory = hostingContext.HostingEnvironment.ContentRootPath;

        config.SetBasePath(currentDirectory)
            .AddJsonFile("settings.json", optional: true, reloadOnChange: true)
            .AddEnvironmentVariables();
        config.Build();
    })
    .ConfigureServices((services) =>
    {
        var retryPolicy = Policy
            .HandleResult<HttpResponseMessage>(response => !response.IsSuccessStatusCode)
            .RetryAsync(3, onRetry: (message, retryCount) =>
            {
                Console.Out.WriteLine("----------------------------------------------------");
                Console.Out.WriteLine($"### RequestMessage: {message.Result.RequestMessage}");
                Console.Out.WriteLine($"### StatusCode: {message.Result.StatusCode}");
                Console.Out.WriteLine($"### ReasonPhrase: {message.Result.ReasonPhrase}");
                Console.Out.WriteLine($"### Retry: {retryCount}");
                Console.Out.WriteLine("----------------------------------------------------");
            });

        services.AddHttpClient<StateCounterHttpClient>()
            .AddPolicyHandler(retryPolicy);
    })
    .Build();

host.Run();

Retry sample code is available on PlayGoKids repository

Polly Await Retry

Depending on the transient fault, contiguous retries may not work, especially if the service is unavailable. It is advisable to try, and wait a bit before retrying it. Polly provides a Wait and Retry policy that allows for multiple contiguous retries with a delay between them.

The UML sequence diagram above shows the case of HTTP 503 (Service Unavailable) responses.

The code example below is extracted from another Azure Function Consumer project that is using the same Typed HttpClient to communicate with an external service. The Polly Await Retry policy is applied to the HttpClient, and it will retry 3 times with an interval of 1000 milliseconds (1 second) if the response is not successful.

Program.cs

using consumer.TypedHttpClient;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Polly;

var host = new HostBuilder()
    .ConfigureFunctionsWorkerDefaults()
    .ConfigureAppConfiguration((hostingContext, config) =>
    {
        var currentDirectory = hostingContext.HostingEnvironment.ContentRootPath;

        config.SetBasePath(currentDirectory)
            .AddJsonFile("settings.json", optional: true, reloadOnChange: true)
            .AddEnvironmentVariables();
        config.Build();
    })
    .ConfigureServices((services) =>
    {
        var retryPolicy = Policy
            .HandleResult<HttpResponseMessage>(response => !response.IsSuccessStatusCode)
            .WaitAndRetryAsync(3,
                _ => TimeSpan.FromMilliseconds(1000),
                onRetry: (message, retryCount) =>
                {
                    Console.Out.WriteLine("----------------------------------------------------");
                    Console.Out.WriteLine($"### RequestMessage: {message.Result.RequestMessage}");
                    Console.Out.WriteLine($"### StatusCode: {message.Result.StatusCode}");
                    Console.Out.WriteLine($"### ReasonPhrase: {message.Result.ReasonPhrase}");
                    Console.Out.WriteLine($"### Retry: {retryCount}");
                    Console.Out.WriteLine("----------------------------------------------------");
                });

        services.AddHttpClient<StateCounterHttpClient>()
            .AddPolicyHandler(retryPolicy);
    })
    .Build();

host.Run();

Await Retry sample code is available on PlayGoKids repository

Polly Await Retry with Jitter strategy

The Polly Await Retry policy can be improved by adding jitter to the delay between retries. This will help to avoid multiple clients retrying at the same time, and it will help to avoid the Thundering Herd³ problem.

The jitter strategy⁴ can help to evenly distribute the load across the clients, and it can be applied to the Polly Await Retry policy as below:

using consumer.TypedHttpClient;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Polly;
using Polly.Contrib.WaitAndRetry;

var host = new HostBuilder()
    .ConfigureFunctionsWorkerDefaults()
    .ConfigureAppConfiguration((hostingContext, config) =>
    {
        var currentDirectory = hostingContext.HostingEnvironment.ContentRootPath;

        config.SetBasePath(currentDirectory)
            .AddJsonFile("settings.json", optional: true, reloadOnChange: true)
            .AddEnvironmentVariables();
        config.Build();
    })
    .ConfigureServices((services) =>
    {
        var delay = Backoff.DecorrelatedJitterBackoffV2(medianFirstRetryDelay: TimeSpan.FromSeconds(1), retryCount: 3);

        var retryPolicy = Policy
            .HandleResult<HttpResponseMessage>(response => !response.IsSuccessStatusCode)
            .WaitAndRetryAsync(delay,
                onRetry: (message, retryCount) =>
                {
                    Console.Out.WriteLine("----------------------------------------------------");
                    Console.Out.WriteLine($"### RequestMessage: {message.Result.RequestMessage}");
                    Console.Out.WriteLine($"### StatusCode: {message.Result.StatusCode}");
                    Console.Out.WriteLine($"### ReasonPhrase: {message.Result.ReasonPhrase}");
                    Console.Out.WriteLine($"### Retry: {retryCount}");
                    Console.Out.WriteLine("----------------------------------------------------");
                });

        services.AddHttpClient<StateCounterHttpClient>()
            .AddPolicyHandler(retryPolicy);
    })
    .Build();

host.Run();

The above implementation leverages Polly.Contrib.WaitAndRetry NuGet package version 1.1.1, and the sample code is available on PlayGoKids repository

Simulating the transient fault

The Azure Function Producer project, which is basically a Counter, simulates transient faults. It is responsible for preserving and incrementing the counter state.

[Function(nameof(Increment))]
public HttpResponseData Increment([HttpTrigger(AuthorizationLevel.Function, "get")] HttpRequestData req)
{
    _logger.LogInformation("Request to increment counter.");

    var isFailureEnabled = _configuration.GetValue<bool>(Constants.FailureEnabled); // variable to control failure injection
    var currentCounter = _tableStorageHelper.GetCounter(Constants.Counter, Constants.PartitionKey, Constants.Row);

    if (isFailureEnabled && currentCounter % 3 == 0)
    {
        const string errorMessage = "Counter is divisible by 3, throwing exception.";

        _logger.LogError(errorMessage);
        throw new Exception(errorMessage);
    }

    var counter = _tableStorageHelper.IncrementCounter(Constants.Counter, Constants.PartitionKey, Constants.Row);

    var response = req.CreateResponse(HttpStatusCode.OK);
    response.Headers.Add("Content-Type", "text/plain; charset=utf-8");

    var message = $"Current counter: {counter}";
    _logger.LogInformation(message);
    response.WriteString(message);

    return response;
}

This Increment function is used to increment a counter, and depending on a condition it simulates a transient fault by throwing an exception when the counter is divisible by 3. The isFailureEnabled variable comes from the settings.json file, and it is also used to control the failure injection.

To be able to run it on your machine, make sure the Table Storage is initialized as below: