Grain Resilience

Retry, circuit-breaker, and timeout strategies for Orleans grain calls, powered by Polly v8.

Current API. The examples wrap ordinary functional API-record calls. The original handle example is retained in Legacy Resilience.

What you’ll learn

• How to retry transient grain failures automatically
• What a timeout over a grain call enforces, and what it cannot cancel
• How to protect a downstream service with a circuit breaker
• How to compose multiple strategies into a single reusable pipeline

---

Overview

GrainResilience is a thin F#-idiomatic wrapper around Polly v8. It lets you wrap any grain call in a resilience pipeline without touching the grain implementation.

open Orleans.FSharp

// Retry a grain call up to 3 times with a 500 ms delay between attempts
let! result =
    GrainResilience.retry<string> 3 (TimeSpan.FromMilliseconds 500) (fun () ->
        grain.HandleMessage(FetchData id))

All helpers work with Task<'T>, keeping the code in your standard task { } expressions.

---

Quickstart

open System
open Orleans.FSharp

// 1. Simple retry — 3 extra attempts with a 200 ms delay
let! inventory =
    GrainResilience.retry<int> 3 (TimeSpan.FromMilliseconds 200) (fun () ->
        inventoryGrain.HandleMessage(GetStock itemId))

// 2. Timeout — a deadline the caller can rely on: after 2 seconds this raises
//    TimeoutRejectedException and abandons the call
//    (see "What the deadline does and does not do" below)
let! price =
    GrainResilience.withTimeout<decimal> (TimeSpan.FromSeconds 2) (fun () ->
        pricingGrain.HandleMessage(GetPrice itemId))

// 3. Full options — timeout + circuit breaker + retry
let opts =
    { GrainResilience.defaultOptions with
        MaxRetryAttempts = 3
        RetryDelay = TimeSpan.FromMilliseconds 100
        Timeout = Some(TimeSpan.FromSeconds 5)
        CircuitBreakerThreshold = Some 5
        CircuitBreakerDuration = Some(TimeSpan.FromSeconds 30) }

let! order =
    GrainResilience.execute<OrderResult> opts (fun () ->
        orderGrain.HandleMessage(PlaceOrder cart))

---

API Reference

ResilienceOptions — configuration record

type ResilienceOptions =
    {
        /// Maximum number of retry attempts after the initial call. Default: 3
        MaxRetryAttempts: int

        /// Delay between retries. Default: 1 second
        RetryDelay: TimeSpan

        /// Open the circuit after this many consecutive failures. None = disabled.
        /// `execute` caches circuit state by the identity of this exact immutable options
        /// instance plus result type — see "Circuit state and pipeline identity" below.
        CircuitBreakerThreshold: int option

        /// How long the circuit stays open before attempting a probe call. None falls back
        /// to 30 s, and is only consulted when CircuitBreakerThreshold is set.
        CircuitBreakerDuration: TimeSpan option

        /// Deadline over the whole protected operation — every attempt plus the delays
        /// between them, not one attempt. None = no timeout.
        /// Read "What the deadline does and does not do" below before relying on it.
        Timeout: TimeSpan option
    }

let defaultOptions: ResilienceOptions =
    {
        MaxRetryAttempts = 3
        RetryDelay = TimeSpan.FromSeconds 1
        CircuitBreakerThreshold = None
        CircuitBreakerDuration = None
        Timeout = None
    }

Two bounds Polly enforces, and it enforces them by throwing System.ComponentModel.DataAnnotations.ValidationException (Polly validates its options with data annotations) when the pipeline is built — that is, from inside the execute / withTimeout call itself, not at configuration time:

• Timeout must be between 10 ms and 24 hours. TimeSpan.Zero is not “no timeout”; use None for that.
• CircuitBreakerThreshold must be at least 2 (it maps to Polly’s MinimumThroughput). Some 1 and Some 0 both throw.

GrainResilience.retry

Retries the grain call up to maxAttempts further times after the initial one, so retry 3 makes at most four calls.

val retry<'T>
    : maxAttempts : int
    -> delay      : TimeSpan
    -> f          : (unit -> Task<'T>)
    -> Task<'T>

let! count =
    GrainResilience.retry<int> 5 (TimeSpan.FromSeconds 1) (fun () ->
        counterGrain.HandleMessage(Increment))

GrainResilience.withTimeout

Applies a Polly timeout strategy around the protected operation, with no retry and no circuit breaker.

val withTimeout<'T>
    : timeout : TimeSpan
    -> f      : (unit -> Task<'T>)
    -> Task<'T>

try
    let! snapshot =
        GrainResilience.withTimeout<Snapshot> (TimeSpan.FromSeconds 3) (fun () ->
            snapshotGrain.HandleMessage(CreateSnapshot))
    processSnapshot snapshot
with :? Polly.Timeout.TimeoutRejectedException ->
    log.Warning("Snapshot timed out — skipping")

What the deadline does and does not do

The deadline always fires for the caller. When it passes, withTimeout raises TimeoutRejectedException and stops waiting — measured, a 100 ms budget over an 800 ms call raises at ~101 ms. (Before 4.0.1 it did not: that same call returned its result after ~810 ms and raised nothing, because the pipeline was handed a task it could only await.)

It does not cancel the call. The protected operation is a unit -> Task<'T>, which takes no CancellationToken, so nothing can interrupt the work itself. The in-flight call is abandoned: it keeps running to completion on the silo, its effects land, and only its result is discarded. A deadline here bounds how long you wait, not what the system does — cancellation without rollback is this library’s documented stance everywhere else too.

For an operation that can honour a token, use withTimeoutCancellable / executeCancellable below: the deadline’s token is handed to the operation, so it can stop rather than be abandoned. Beyond that, the other real deadlines available are Orleans’ own — SiloMessagingOptions.ResponseTimeout / ClientMessagingOptions.ResponseTimeout (30 seconds by default), and FunctionalGrainRef.callCancellable from the functional grain runtime, whose token reaches the handler through context.cancellationToken.

GrainResilience.withTimeoutCancellable

The same deadline, handed to an operation that takes the token.

val withTimeoutCancellable<'T>
    : timeout : TimeSpan
    -> f      : (CancellationToken -> Task<'T>)
    -> Task<'T>

let! rows =
    GrainResilience.withTimeoutCancellable<Row list> (TimeSpan.FromSeconds 3) (fun ct ->
        reportGrain.RunQuery(query, ct))

The token is cancelled when the deadline passes. An OperationCanceledException raised in response is reported to the caller as TimeoutRejectedException, so both entry points fail the same way. An operation that ignores the token is still abandoned exactly as under withTimeout, so the deadline holds either way.

To honour a caller’s own token as well, link it inside f:

GrainResilience.withTimeoutCancellable timeout (fun deadlineToken ->
    use linked = CancellationTokenSource.CreateLinkedTokenSource(deadlineToken, context.cancellationToken)
    grain.RunQuery(query, linked.Token))

A cancel that arrives through your own token surfaces as OperationCanceledException, not as TimeoutRejectedException — Polly only rebrands a cancellation it caused itself.

GrainResilience.execute

Full-options entry point. Compose timeout, circuit breaker, and retry in one call.

val execute<'T>
    : options : ResilienceOptions
    -> f      : (unit -> Task<'T>)
    -> Task<'T>

let myOpts =
    { GrainResilience.defaultOptions with
        MaxRetryAttempts = 2
        Timeout = Some(TimeSpan.FromSeconds 10) }

let! result = GrainResilience.execute<string> myOpts (fun () -> grain.HandleMessage cmd)

Every attempt re-invokes f, so MaxRetryAttempts = 2 calls it at most three times. execute caches its pipeline by the object identity of the exact immutable ResilienceOptions value and by 'T: reusing myOpts for the same result type reuses the pipeline and shares circuit state.

GrainResilience.executeCancellable

execute for an operation that takes the deadline’s token — the full-options counterpart of withTimeoutCancellable. It uses the same options-instance/result-type pipeline cache as execute.

val executeCancellable<'T>
    : options : ResilienceOptions
    -> f      : (CancellationToken -> Task<'T>)
    -> Task<'T>

let! result =
    GrainResilience.executeCancellable<Report> myOpts (fun ct ->
        reportGrain.Build(spec, ct))

GrainResilience.buildPipeline

Creates a fresh ResiliencePipeline<'T> from options on every call. This is the explicit pipeline construction API: retain and reuse the returned object when calls should share circuit state, or call buildPipeline again when they should be isolated.

val buildPipeline<'T> : options : ResilienceOptions -> ResiliencePipeline<'T>

let pipeline = GrainResilience.buildPipeline<int> myOpts

// Reuse the same pipeline object many times
let! r1 = pipeline.ExecuteAsync(fun _ -> ValueTask<int>(grain1.HandleMessage cmd)).AsTask()
let! r2 = pipeline.ExecuteAsync(fun _ -> ValueTask<int>(grain2.HandleMessage cmd)).AsTask()

GrainResilience.circuitBreaker

Creates a standalone, non-generic ResiliencePipeline backed only by a circuit breaker. Because the circuit-state is held inside the returned object, you should keep it as a long-lived value (e.g., a let binding at the service scope).

val circuitBreaker
    : threshold     : int
    -> breakDuration : TimeSpan
    -> ResiliencePipeline

// Open after 5 failures; stay open for 30 seconds
let private cb = GrainResilience.circuitBreaker 5 (TimeSpan.FromSeconds 30)

member _.CallExternalService(cmd) =
    task {
        try
            return! cb.ExecuteAsync(fun _ -> ValueTask<_>(grain.HandleMessage cmd)).AsTask()
        with :? Polly.CircuitBreaker.BrokenCircuitException ->
            return Error "Service unavailable"
    }

---

Strategy composition order

When you use execute with multiple strategies enabled, they are layered outer → inner:

request
  → [Timeout]          ← outermost; cancels everything inside if deadline exceeded
    → [CircuitBreaker] ← trips on consecutive failures; short-circuits when open
      → [Retry]        ← innermost; retries on exceptions
        → grain call

This means:

• The timeout is a deadline over the whole attempt sequence, not over one attempt: it covers every attempt and every delay between them. One attempt that hangs therefore consumes the entire budget and the retries never happen — that is the deliberate trade of a total deadline, and it is the guarantee the caller can actually reason about (see What the deadline does and does not do).

• For a per-attempt deadline, nest the two: retry on the outside, withTimeout inside the function it retries.

  GrainResilience.retry<int> 3 (TimeSpan.FromMilliseconds 200) (fun () ->
      GrainResilience.withTimeout<int> (TimeSpan.FromSeconds 1) (fun () ->
          grain.HandleMessage cmd))

• The circuit breaker sits between them, so it sees one outcome per execution — the retry strategy’s final verdict, not each attempt.

• A single Polly TimeoutRejectedException or BrokenCircuitException bypasses the retry.

Circuit state and pipeline identity

Circuit-breaker state lives in a pipeline. execute and executeCancellable cache that pipeline by two keys: the object identity of the exact immutable ResilienceOptions instance, and the result type 'T. Reusing one options binding with the same result type therefore shares circuit state across calls. Independently constructed options records remain isolated even when all their fields are structurally equal; changing the result type also selects a separate pipeline.

This makes the sharing boundary explicit:

// Reuse this exact immutable instance: execute<int> calls share one circuit.
let sharedOptions =
    { GrainResilience.defaultOptions with
        MaxRetryAttempts = 0
        CircuitBreakerThreshold = Some 5
        CircuitBreakerDuration = Some(TimeSpan.FromSeconds 30) }

let callShared f = GrainResilience.execute<int> sharedOptions f

// A distinct record instance, even with equal fields, owns an isolated circuit.
let isolatedOptions = { sharedOptions with MaxRetryAttempts = sharedOptions.MaxRetryAttempts }

For direct control, build one pipeline and reuse it:

// Service-scoped: one object, one circuit
let private pipeline =
    GrainResilience.buildPipeline<int>
        { GrainResilience.defaultOptions with
            CircuitBreakerThreshold = Some 5
            CircuitBreakerDuration = Some(TimeSpan.FromSeconds 30) }

member _.Query(cmd) =
    pipeline.ExecuteAsync(fun _ -> ValueTask<_>(grain.HandleMessage cmd)).AsTask()

buildPipeline itself always returns a fresh pipeline; only reuse of the returned object shares its state. Alternatively, use GrainResilience.circuitBreaker, which is a standalone non-generic pipeline. Tests pin all three boundaries: reused options identity shares under execute, a separate options instance isolates, and each buildPipeline call is fresh.

---

Patterns

Retry transient connectivity errors

let! data =
    GrainResilience.retry<Data> 3 (TimeSpan.FromMilliseconds 200) (fun () ->
        dataGrain.HandleMessage(Fetch key))

Bound the time spent retrying

// The timeout caps the retry sequence, including the delays between attempts.
let bounded =
    { GrainResilience.defaultOptions with
        MaxRetryAttempts = 5
        RetryDelay = TimeSpan.FromMilliseconds 500
        Timeout = Some(TimeSpan.FromSeconds 2) }

let! result = GrainResilience.execute<Result> bounded (fun () -> flakyGrain.HandleMessage query)

After two seconds the caller gets TimeoutRejectedException whatever the sequence is doing — mid-attempt or mid-delay. What it does not do is stop the attempt that was in flight; see What the deadline does and does not do.

Protect a shared downstream resource

// Service-scoped — shared circuit state across all calls
let cb = GrainResilience.circuitBreaker 10 (TimeSpan.FromMinutes 1)

member _.Query(cmd) =
    cb.ExecuteAsync(fun _ -> ValueTask<_>(grain.HandleMessage cmd)).AsTask()

Full production resilience

let productionOpts =
    { MaxRetryAttempts       = 3
      RetryDelay             = TimeSpan.FromMilliseconds 500
      CircuitBreakerThreshold = Some 10
      CircuitBreakerDuration  = Some(TimeSpan.FromSeconds 60)
      Timeout                = Some(TimeSpan.FromSeconds 15) }

let! response =
    GrainResilience.execute<ApiResponse> productionOpts (fun () ->
        apiGrain.HandleMessage(ApiRequest payload))

Wrapping a functional API call

let! price =
    GrainResilience.execute options (fun () -> pricingApi.getPrice itemId)

Testing resilience

In unit tests, you can drive failures by throwing exceptions inside a closure without touching any real grain:

let mutable attempts = 0

let! result =
    GrainResilience.retry<int> 3 TimeSpan.Zero (fun () ->
        task {
            attempts <- attempts + 1
            if attempts < 3 then failwith "transient"
            return 42
        })

test <@ result = 42 @>
test <@ attempts = 3 @>

To test a deadline, do not assert that it fired within some number of milliseconds — that is a wall-clock budget, and a loaded machine will fail it eventually. Gate the protected call on a TaskCompletionSource that the test never releases inside the timed region, so the call cannot finish on its own, and assert which task completed first under a net so generous it carries no information about speed:

let gate = TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously)

let guarded =
    task {
        try
            let! _ = GrainResilience.withTimeout (TimeSpan.FromMilliseconds 50.0) (fun () -> gate.Task)
            return null :> exn
        with ex -> return ex
    }

let! finished = Task.WhenAny(guarded, Task.Delay(TimeSpan.FromSeconds 30.0))
test <@ Object.ReferenceEquals(finished, guarded) @>
let! ex = guarded
test <@ ex :? Polly.Timeout.TimeoutRejectedException @>

For integration tests with a real TestCluster, use a grain that tracks its own call count and fails on the first N calls — see flakyGrain (over FlakyState / FlakyCommand) in tests/Orleans.FSharp.Integration/ClusterFixture.fs, driven by GrainResilienceIntegrationTests.fs.

---

Package dependency

GrainResilience is in the Orleans.FSharp core package. It adds a dependency on Polly 8.x, which is already bundled — no additional NuGet reference is required.