Multithreading – garbage collection in Perl threads

The question is curiosity because one of the following two items works

I'm using @ L_ 403_ 0 @:: magick resizes a large number of photos To save some time, I process each photo in my own thread and use semaphores to limit the number of threads working at the same time Initially, I allowed each thread to run once, but the script will quickly allocate 3.5 GB for all photos (I only have 2GB available), and because all photos are switched to disk, the script will be five times slower than normal

The semaphore version code of the working is as follows:

use threads;
use Thread::Semaphore;
use Image::Magick;

my $s = Thread::Semaphore->new(4);
foreach ( @photos ) {
    threads->create( \&launch_thread,$s );
}
foreach my $thr ( reverse threads->list() ) {
    $thr->join();
}

sub launch_thread {
    my $s = shift;
    $s->down();
    my $image = Image::Magick->new();

    # do memory-heavy work here

    $s->up();
}

This quickly allocates 500MB and works very well without requiring more (threads are connected in reverse order to form a point.)

I wanted to know if it was possible to start 80 threads at the same time and block most threads, so I changed my script to block the main thread:

my $s = Thread::Semaphore->new(4);
foreach ( @photos ) {
    $s->down();
    threads->create( \&launch_thread,$s );
}
foreach my $thr ( threads->list() ) {
    $thr->join();
}

sub launch_thread {
    my $s = shift;
    my $image = Image::Magick->new();

    # do memory-heavy work here

    $s->up();
}

This version starts normal, but gradually accumulates the 3.5GB space used by the original version It is faster than running all threads at once, but still a little slower than blocking threads

My first guess is that the memory used by the thread will not be released before calling join (), and because it is the blocked main thread, no thread will be released until they are all allocated However, in the first working version, threads passed protection in more or less random order, but connected in the opposite order If my guess is correct, then in addition to the four running threads, you should wait for any time to join (), and this version should be slower

So why are the two versions so different?

Solution

You do not need to create more than four threads A major benefit is that this means a reduction of 76 copies of the Perl interpreter Moreover, since all threads are completed more or less at the same time, the harvesting sequence is quite meaningless

use threads;
use Thread::Queue qw( );
use Image::Magick qw( );

use constant NUM_WORKERS => 4;

sub process {
   my ($photo) = @_;
   ...
}

{
   my $request_q = Thread::Queue->new();

   my @threads;
   for (1..NUM_WORKERS) {
       push @threads,async {
          while (my $photo = $request_q->dequeue()) {
             process($photo);
          }
       };
   }

   $request_q->enqueue($_) for @photos;
   $request_q->enqueue(undef) for 1..NUM_THREADS;
   $_->join() for @threads;
}