Multithreading – consumer – producer: pauses the consumer if memory usage exceeds a certain threshold

I have a situation where my main thread (producer) allocates a large amount of memory for a task on the heap, some work on the buffer, and then provides a buffer for the worker thread (consumer) for further processing (this will first compress the data) and then write it to disk) Once the worker thread completes its work, it releases the memory obtained by the producer for the task

However, it is possible that my main thread allocated too much memory, so my system began to swap other programs to disk to meet the memory requirements As the disk becomes busy, the worker thread finds it difficult to write to the disk (eventually freeing any memory), while the producer continues to allocate more memory for other tasks This will eventually destroy the performance of my system

What can be a good design for this problem?

In addition, if you pause the main thread by pre calculating the memory requirements, you can choose in advance how to obtain reliable numbers?

Solution

Possible design options

>Single producer multiple consumers prevents queues between producers and workers > atomic tasks each worker in the inbox is busy rotating / blocking tasks in pull and producer cycles and cannot be published (I think herb Sutter used this design in one of his concurrent Lectures)

In terms of memory allocation, certainty is beneficial, even if all content is pre allocated at startup Strictly speaking, it is not always feasible or practical, so the combination of fixed / dynamic resizing and start / run-time allocation usually occurs in any non trivial system