Debugging 'PHP Fatal error: Allowed memory size exhausted' Correctly

Every backend developer eventually encounters the dreaded "White Screen of Death" or a crashed background worker with a specific log entry:

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes)

The immediate reaction is almost always the same: open php.ini or the specific script and crank up the memory_limit. While changing the limit from 128M to 512M (or worse, -1) solves the immediate crash, it is rarely the correct engineering solution.

Increasing the memory limit without understanding the root cause is technical debt. It turns an O(1) memory operation into an O(n) operation that will inevitably crash again as your dataset grows.

This guide explores why PHP runs out of memory, why arrays are expensive, and how to rewrite data-heavy processes using Generators and Streams to keep memory usage flat, regardless of input size.

The Root Cause: Why PHP Eats RAM

To fix memory exhaustion, you must understand how PHP manages variables. PHP uses a reference-counting memory management system with a Garbage Collector (GC) to handle circular references.

The Cost of Arrays

The most common culprit for memory exhaustion is loading a large database result set into an array using methods like fetchAll().

In C (the language PHP is written in), an array is not just a block of memory. It is a hash table (Bucket). Every element in a PHP array requires a zval (Zend Value) structure, bucket overhead, and the actual data.

Consequently, a 100MB CSV file might require 400MB+ of RAM to represent as a PHP array. If your script processes 10,000 rows today, it works. When the business grows to 1,000,000 rows, a memory_limit increase won't save you—you will hit physical hardware limits.

Buffered vs. Unbuffered Queries

By default, most PHP database drivers (like PDO/MySQL) use buffered queries. When you execute a SELECT statement, the driver transparently fetches every single row from the database server and stores it in the PHP process's RAM before your code even iterates over the first row.

This reduces network chatter but destroys memory efficiency for large datasets.

The Fix: Generators and Unbuffered Queries

The solution is to decouple memory usage from dataset size. We achieve this by:

Disabling Query Buffering: Fetching one row at a time from the database socket.
Using Generators (yield): Passing data through the application one item at a time instead of building a monolithic array.
Streaming Output: Writing directly to the output buffer or file handle.

The Scenario

Imagine we need to export 500,000 user records to a CSV file.

The "Crashy" Approach (Anti-Pattern)

Do not use this code.

// BAD: Loads everything into RAM
$stmt = $pdo->query("SELECT * FROM users");
$users = $stmt->fetchAll(PDO::FETCH_ASSOC); // <--- OOM Crash happens here

$csv = '';
foreach ($users as $user) {
    $csv .= implode(',', $user) . "\n"; // <--- String concatenation doubles memory usage
}
file_put_contents('export.csv', $csv);

The Modern, Memory-Safe Solution

Below is a robust, production-ready implementation using PHP 8.2+ features. This script will run with memory_limit = 2M regardless of whether you export 500 rows or 5 million.

<?php

declare(strict_types=1);

/**
 * Modern implementation of a memory-safe CSV exporter.
 */
class LargeDatasetExporter
{
    private PDO $pdo;

    public function __construct(
        private readonly string $dbDsn,
        private readonly string $dbUser,
        private readonly string $dbPass
    ) {
        // 1. Establish connection
        $this->pdo = new PDO($this->dbDsn, $this->dbUser, $this->dbPass, [
            PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
            PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC,
        ]);
    }

    /**
     * Generator function that yields rows one by one.
     * This keeps only ONE row in memory at a time.
     */
    public function getUsersIterator(): Generator
    {
        // 2. Disable Buffering for MySQL
        // This forces the driver to read from the stream rather than loading all results.
        $this->pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);

        $stmt = $this->pdo->query("SELECT id, name, email, created_at FROM users");

        while ($row = $stmt->fetch()) {
            yield $row;
        }
        
        // Optional: Close cursor to free database resources immediately
        $stmt->closeCursor();
    }

    /**
     * Streams data directly to a file handle.
     */
    public function exportToCsv(string $outputPath): void
    {
        // Open file stream for writing
        $fileHandle = fopen($outputPath, 'w');
        
        if ($fileHandle === false) {
            throw new RuntimeException("Unable to open file for writing: $outputPath");
        }

        $iterator = $this->getUsersIterator();
        $headersWritten = false;
        $count = 0;

        foreach ($iterator as $row) {
            // Write headers dynamically based on first row keys
            if (!$headersWritten) {
                fputcsv($fileHandle, array_keys($row));
                $headersWritten = true;
            }

            // Write the row immediately to disk
            fputcsv($fileHandle, $row);
            
            // 3. Periodic Garbage Collection (Optional optimization for long-running scripts)
            $count++;
            if ($count % 1000 === 0) {
                // Forces cyclic garbage collection if any circular refs were created
                gc_collect_cycles(); 
            }
        }

        fclose($fileHandle);
        
        echo "Successfully exported $count rows using minimal RAM." . PHP_EOL;
        echo "Peak Memory Usage: " . (memory_get_peak_usage(true) / 1024 / 1024) . " MB" . PHP_EOL;
    }
}

// Usage Example
try {
    $exporter = new LargeDatasetExporter(
        'mysql:host=127.0.0.1;dbname=production_db',
        'readonly_user',
        'secure_pass'
    );
    
    $exporter->exportToCsv(__DIR__ . '/users_export.csv');
    
} catch (Exception $e) {
    fwrite(STDERR, "Error: " . $e->getMessage() . PHP_EOL);
    exit(1);
}

Deep Dive: Why This Works

1. The Generator (`yield`)

The magic keyword here is yield. In the getUsersIterator method, the function does not return an array. Instead, it returns an instance of the Generator class.

When the foreach loop in exportToCsv requests the next item, the getUsersIterator function resumes execution, fetches one row, yields it, and then pauses. The previous row is discarded (and cleared from memory) before the next one is fetched.

2. Unbuffered Queries

The line $this->pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false); is critical for MySQL.

Without this, PHP acts as a buffer. If your query returns 5GB of data, the MySQL driver attempts to download 5GB into PHP's RAM before giving your code control. By disabling buffering, PHP opens a cursor on the MySQL server and pulls data packet-by-packet via the network socket only when requested.

3. Stream-to-Disk (`fputcsv`)

We avoid string concatenation ($csv .= ...). Concatenating strings in a loop creates a new string in memory for every iteration, which is extremely expensive. fputcsv writes directly to the file resource pointer, flushing data to the OS filesystem cache rather than holding it in PHP's user-space memory.

Common Pitfalls and Edge Cases

Even with Generators, you can still leak memory. Watch out for these subtle issues.

Circular References

If your loop instantiates objects that reference each other (e.g., Parent has a Child, Child has a property pointing back to Parent), PHP's simple reference counter cannot clear them when they go out of scope. They become "orphaned" in memory.

Solution: Although PHP 5.3+ has a garbage collector that eventually cleans these up, explicitly calling gc_collect_cycles() every few thousand iterations (as shown in the code example) ensures memory is reclaimed during long-running processes.

ORM Hydration Overhead

If you are using an ORM like Eloquent (Laravel) or Doctrine, be very careful.

Eloquent: User::all() creates a model instance for every row. This is massive overhead. Use User::cursor() (which uses generators internally) or DB::table('users')->cursor() for raw query performance.
Doctrine: Use iterate() instead of findAll().

Logging Inside Loops

Logging libraries (like Monolog) often keep a buffer of logs in memory to batch-write them or format them with stack traces.

Solution: When processing large datasets, avoid logging full objects inside the loop. Log only IDs or summaries, and ensure your logger is configured to flush to disk immediately rather than buffering.

Conclusion

The "Allowed memory size exhausted" error is rarely a signal that you need more RAM. It is a signal that your application is trying to load a dataset into memory that exceeds the physical constraints of the machine.

By switching from eager loading (fetchAll) to lazy evaluation (yield) and unbuffered database queries, you can process gigabytes of data with megabytes of RAM. This makes your application faster, cheaper to host, and significantly more stable under load.

Programming Tutorials

Search This Blog