Programming Tutorials

  The error is all too familiar. Your ETL script runs perfectly on your local machine with a 50MB sample file. You deploy it to production to process a 10GB CSV dump, and 30 seconds later, the process dies: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory This is not a memory leak in the traditional sense. It is a flow control failure. Specifically, your application is reading data from the source faster than it can write it to the destination. The Root Cause: Unbounded Buffering In Node.js, streams are designed to transport data. However, data takes up memory while it is being transported. When you read from a fast source (like  fs.createReadStream  reading from an SSD) and write to a slow destination (like a database insert over the network or  fs.createWriteStream  to a slow disk), a bottleneck forms. Internally,  Writable  streams have a  highWaterMark  property (defaulting to 16KB)....