Skip to main content

Rust Async Patterns: Solving 'Cannot Move Out of Shared Reference' with Pinning

 One of the most notoriously difficult hurdles in async Rust—specifically when implementing manual Future types or middleware—is the compilation error: cannot move out of ... which is behind a shared reference or cannot borrow data in a '&' reference as mutable.

When you encounter this inside a Future::poll implementation, it is usually a symptom of a misunderstood memory model regarding Pin. You are attempting to access a field of a struct that is structurally pinned, but you are not projecting that pin correctly to the field.

This post dissects why naive field access fails in async contexts and demonstrates the correct implementation using structural pinning.

The Root Cause: Pinning and Memory Stability

To understand the error, we must look at the signature of the Future trait:

pub trait Future {
    type Output;
    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
}

The self argument is not &mut Self; it is Pin<&mut Self>.

Async blocks in Rust compile down to state machines. These state machines are often self-referential—they may hold pointers to their own fields across .await points. If such a struct were moved in memory (e.g., via mem::swap or passing by value), those internal pointers would dangle, causing undefined behavior.

Pin<P> wraps a pointer type P and prevents the value pointed to from being moved unless that type implements Unpin.

The Problem Scenario

When you implement a manual Future wrapper (e.g., a timeout, a logger, or a retry mechanism), you typically hold a child future inside your struct. To drive the wrapper, you must drive the child.

// ❌ THIS WILL FAIL TO COMPILE
impl<F: Future> Future for MyWrapper<F> {
    type Output = F::Output;

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        // Error: `self` is a Pin<&mut Self>. 
        // Rust prevents you from getting a mutable reference `&mut F` 
        // because `F` might be !Unpin.
        self.child.poll(cx) 
    }
}

The compiler blocks this because Pin<&mut Self> does not automatically grant Pin<&mut Field>. If it gave you a raw &mut Field, you could swap the field out, breaking the pinning invariant of the parent. To fix this, we need Pin Projection.

The Solution: Structural Pinning with pin-project-lite

While you can implement unsafe manual pointer projection, it is error-prone and discouraged for production code. The industry standard is to use pin-project-lite (a zero-dependency version of pin-project) to safely generate the projection code.

This example creates a TimeoutFuture that wraps an inner future. It requires accessing the inner future to poll it, while maintaining valid pinning guarantees.

Step 1: Dependencies

Add this to your Cargo.toml:

[dependencies]
pin-project-lite = "0.2"
tokio = { version = "1", features = ["full"] } // For runtime example

Step 2: The Implementation

We use the pin_project! macro to define the struct. This macro generates a project() method on Pin<&mut Self> that returns a projection struct containing Pin<&mut Field> for pinned fields.

use std::future::Future;
use std::pin::Pin;
use std::task::{Context, Poll};
use std::time::Duration;
use tokio::time::{Sleep, Instant};
use pin_project_lite::pin_project;

// Use the macro to define the struct.
// This allows us to opt-in to pinning for specific fields.
pin_project! {
    struct TimeoutFuture<F> {
        // We need to poll this, so it must be pinned.
        #[pin]
        inner: F,
        
        // We need to poll this too.
        #[pin]
        timer: Sleep,
        
        // Simple data does not need pinning.
        duration: Duration,
    }
}

impl<F: Future> Future for TimeoutFuture<F> {
    type Output = Result<F::Output, &'static str>;

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        // 1. PROJECT THE PIN
        // The `.project()` method consumes `Pin<&mut Self>` and returns
        // a new struct (TimeoutFutureProj) where fields marked #[pin] 
        // are now `Pin<&mut T>`, and unpinned fields are `&mut T`.
        let this = self.project();

        // 2. POLL THE INNER FUTURE
        // `this.inner` is now `Pin<&mut F>`, which matches the signature required by `poll`.
        if let Poll::Ready(output) = this.inner.poll(cx) {
            return Poll::Ready(Ok(output));
        }

        // 3. POLL THE TIMER
        // `this.timer` is `Pin<&mut Sleep>`.
        if let Poll::Ready(()) = this.timer.poll(cx) {
            return Poll::Ready(Err("Future timed out"));
        }

        Poll::Pending
    }
}

// Usage Example
#[tokio::main]
async fn main() {
    let slow_task = async {
        tokio::time::sleep(Duration::from_millis(200)).await;
        42
    };

    let timeout_task = TimeoutFuture {
        inner: slow_task,
        timer: tokio::time::sleep_until(Instant::now() + Duration::from_millis(100)),
        duration: Duration::from_millis(100),
    };

    match timeout_task.await {
        Ok(val) => println!("Task finished with: {}", val),
        Err(e) => println!("Task failed: {}", e),
    }
}

The Explanation: How Projection Works

The project() method is the key. Without it, you are stuck looking at the Pin wrapper from the outside.

When you call self.project(), the library performs a transformation roughly equivalent to this (simplified for clarity):

// Conceptually what happens
struct TimeoutFutureProj<'a, F> {
    inner: Pin<&'a mut F>,      // Pinned!
    timer: Pin<&'a mut Sleep>,  // Pinned!
    duration: &'a mut Duration, // Not pinned (standard mutable ref)
}

impl<F> TimeoutFuture<F> {
    fn project<'a>(self: Pin<&'a mut Self>) -> TimeoutFutureProj<'a, F> {
        unsafe {
            let this = self.get_unchecked_mut();
            TimeoutFutureProj {
                // UNSAFE: We promise strictly that `inner` will not be moved
                inner: Pin::new_unchecked(&mut this.inner),
                timer: Pin::new_unchecked(&mut this.timer),
                duration: &mut this.duration,
            }
        }
    }
}

Why implies "Structural Pinning"

When we mark a field as #[pin], we are asserting Structural Pinning. This means the pinning guarantee of the parent (TimeoutFuture) is transitively applied to the child (inner).

  1. If the parent is pinned: The child is pinned.
  2. If the parent is never moved: The child is never moved.
  3. Dropping: If the parent is dropped, the child is dropped (without being moved).

By converting Pin<&mut Self> into Pin<&mut Field>, we satisfy the Future::poll requirement of the child.

Why Unpin matters

If F implemented Unpin (like u32 or a standard Box<T>), pinning would be irrelevant. You could freely get &mut F from Pin<&mut F>. However, async/await generates anonymous types that are explicitly !Unpin (Not Unpin). This is why manual projection is mandatory for generic middleware—you must assume the wrapped future is self-referential and immobile.

Conclusion

The error "cannot move out of ... behind a shared reference" in async Rust is a safeguard against invalidating self-referential pointers.

To solve it:

  1. Do not attempt to bypass Pin by dereferencing it blindly.
  2. Use pin-project-lite to define your structures.
  3. Mark fields that are Futures (or Streams) with #[pin].
  4. Call .project() inside poll to get a view of your fields that honors their pinning requirements.

This pattern ensures safe, zero-cost abstractions over the async state machine without risking undefined behavior.

Labels: