Multithreading – Perl threads slowly consume memory

I'm running a Perl server with 10 threads They will never be broken before the program exits, but this is my intention to run as long as possible, so that's why this is my problem A thread processes a simple task multiple times When I start the server and all threads start, I see I have 288.30 MB Free After several iterations per thread, it reported 285.96 MB Free It's not that bad, but in these iterations, just some stack space is allocated or something But after 15 minutes, the available memory dropped to 248.24 MB! What happened to my memory? Now, interestingly, it is a plateau It continues to consume slowly, but not as fast as the first time I thought it might be my fault, so I tried to recheck the scope of all variables and even define them all at the end of the thread loop

I print out the available space after each thread iteration, so I can read it slowly What's interesting now is that it doesn't decrease every time Sometimes, the free memory after the loop will remain unchanged

I use Perl 5.8.0 built from source code on Linux 2.6 eight

Does anyone have any ideas or even suggestions that may have caused this? I am considering upgrading my Perl to a later version to eliminate memory leaks in the Perl kernel

Update: could this be a thread stack size issue? I can allocate more memory for the stack than I need When I create my thread, I will not change the setting from the default value Should I? Thread doc indicates that the default value is usually 16MB, depending on the system sixteen × 10 threads = 160MB – > this may be the culprit reflection?

Update: I built and installed Perl 5.12 1 and rebuild the module and everything Now run the script for about an hour, which I noticed Memory usage is now manageable, but not ideal

After I started, my thread seemed to drop a little From ~ 60-66mb to my 10 threads ~ ~ 45-50mb. > After some iterations, their usage has increased by a total of 3MB (roughly the same as before). > So far, my expectation is All the memory is for generation, and then just a little bit of variables I use in the thread This is the part I don't like After running for about 10 minutes, I lost 65mb again! Why? If it has been iterated several times and only 3MB, why allocate? > At this point, it has been running for an hour and a half, and has not used the additional 65mb, and there are 84Mb! > It slowly requires more memory, but strangely, the amount of available memory does not decrease with each iteration I print out the free memory before and after each iteration, which will remain for a period of time or hover - around a certain amount of time, and then suddenly change 5-10mb I can't leave this running for more than a day or two because it starts to approach 80 / 90% of the available memory

Any other ideas? Can I try anything? I don't know all my variables anymore

Update: I really want to continue recompiling Perl and glibc as a last resort, because I found some reports that in some styles of Linux, it will segfault So since my last release, I've further explored the possibility of loops in my hash I didn't find it, so I spent the last few days analyzing my subroutine and buffering anything used in another iteration A lot of new things are recreated every time, and Perl won't clean them up even if I explicitly delete them all So if I don't cooperate, I won't destroy it You will see if caching my objects is helpful Memory usage statistics will be published later

Update: Well, it's strange that even if I cache my data and use it again, the memory increases at the same rate It starts higher because I'm caching, but over time, although it mainly uses my cached objects It's confusing. Guess it's time to give glibc a try? Or is this just a disadvantage of choosing Perl, restarting the server every few days

Update: tried no cache, no glibc, again Work for a while, a few hours, and then start growing Just want you to see a chart http://tinypic.com/r/311nc08/3 http://i32.tinypic.com/311nc08.jpg

Update: This is a log of available memory before and after each thread in a minute Perhaps this can help people better understand the problem This seems to be stable, and then eat more memories like this every time I lost almost 40 MB here!

[9:8:30,Fri Jul 23,2010] [0] Memory usage at end thread 1: 253.812736MB (obj cache: 136)
[9:8:30,2010] [0] Memory usage at idle thread 1: 253.812736MB (obj cache: 136)
[9:8:34,2010] [204] Sending data to thread
[9:8:34,2010] [0] 3 - Creating a new obj
[9:8:34,2010] [206] Sending data to thread
[9:8:34,2010] [0] 4 - Creating a new obj
[9:8:35,2010] [0] Memory usage at end thread 3: 253.812736MB (obj cache: 136)
[9:8:35,2010] [0] Memory usage at idle thread 3: 253.812736MB (obj cache: 136)
[9:8:35,2010] [0] Memory usage at end thread 4: 253.812736MB (obj cache: 136)
[9:8:35,2010] [0] Memory usage at idle thread 4: 253.812736MB (obj cache: 136)
[9:8:41,2010] [225] Sending data to thread
[9:8:41,2010] [0] 2 - Creating a new obj
[9:8:42,2010] [0] Memory usage at end thread 2: 253.681664MB (obj cache: 136)
[9:8:42,2010] [0] Memory usage at idle thread 2: 253.681664MB (obj cache: 136)
[9:8:47,2010] [243] Sending data to thread
[9:8:47,2010] [0] 1 - Creating a new obj
[9:8:48,2010] [0] Memory usage at end thread 1: 253.935616MB (obj cache: 136)
[9:8:48,2010] [0] Memory usage at idle thread 1: 253.935616MB (obj cache: 136)
[9:9:1,2010] [277] Sending data to thread
[9:9:1,2010] [0] 3 - Creating a new obj
[9:9:2,2010] [280] Sending data to thread
[9:9:2,2010] [0] 4 - Creating a new obj
[9:9:2,2010] [0] Memory usage at end thread 3: 253.935616MB (obj cache: 136)
[9:9:2,2010] [0] Memory usage at idle thread 3: 253.935616MB (obj cache: 136)
[9:9:3,2010] [283] Sending data to thread
[9:9:3,2010] [0] 2 - Creating a new obj
[9:9:4,2010] [284] Sending data to thread
[9:9:4,2010] [0] 1 - Creating a new obj
[9:9:4,2010] [0] Memory usage at end thread 2: 253.935616MB (obj cache: 136)
[9:9:4,2010] [0] Memory usage at idle thread 2: 253.935616MB (obj cache: 136)
[9:9:5,2010] [287] Sending data to thread
[9:9:5,2010] [0] 3 - Creating a new obj
[9:9:5,2010] [0] Memory usage at end thread 4: 253.93152MB (obj cache: 136)
[9:9:5,2010] [0] Memory usage at idle thread 4: 253.93152MB (obj cache: 136)
[9:9:6,2010] [290] Sending data to thread
[9:9:6,2010] [0] 2 - Creating a new obj
[9:9:7,2010] [0] Memory usage at end thread 3: 253.804544MB (obj cache: 136)
[9:9:7,2010] [0] Memory usage at idle thread 3: 253.804544MB (obj cache: 136)
[9:9:7,2010] [0] Memory usage at end thread 1: 253.804544MB (obj cache: 136)
[9:9:7,2010] [0] Memory usage at idle thread 1: 253.804544MB (obj cache: 136)
[9:9:9,2010] [0] 4 - Creating a new obj
[9:9:9,2010] [301] Sending data to thread
[9:9:9,2010] [0] 3 - Creating a new obj
[9:9:9,2010] [302] Sending data to thread
[9:9:9,2010] [0] 1 - Creating a new obj
[9:9:10,2010] [0] 3 - Creating a new obj
[9:9:11,2010] [0] Memory usage at end thread 4: 253.93152MB (obj cache: 136)
[9:9:11,2010] [0] Memory usage at idle thread 4: 253.93152MB (obj cache: 136)
[9:9:12,2010] [308] Sending data to thread
[9:9:12,2010] [0] 4 - Creating a new obj
[9:9:13,2010] [0] Memory usage at end thread 1: 253.804544MB (obj cache: 136)
[9:9:13,2010] [0] Memory usage at idle thread 1: 253.804544MB (obj cache: 136)
[9:9:14,2010] [0] Memory usage at end thread 4: 253.804544MB (obj cache: 136)
[9:9:14,2010] [0] Memory usage at idle thread 4: 253.804544MB (obj cache: 136)
[9:9:14,2010] [0] Memory usage at end thread 3: 253.93152MB (obj cache: 136)
[9:9:14,2010] [0] Memory usage at idle thread 3: 253.93152MB (obj cache: 136)
[9:9:15,2010] [313] Sending data to thread
[9:9:15,2010] [0] 1 - Creating a new obj
[9:9:16,2010] [0] Memory usage at end thread 2: 214.482944MB (obj cache: 136)
[9:9:16,2010] [0] Memory usage at idle thread 2: 214.482944MB (obj cache: 136)
[9:9:16,2010] [315] Sending data to thread
[9:9:16,2010] [0] 4 - Creating a new obj
[9:9:17,2010] [0] Memory usage at end thread 1: 214.355968MB (obj cache: 136)
[9:9:17,2010] [0] Memory usage at idle thread 1: 214.355968MB (obj cache: 136)
[9:9:18,2010] [316] Sending data to thread
[9:9:18,2010] [0] 3 - Creating a new obj
[9:9:18,2010] [317] Sending data to thread
[9:9:18,2010] [0] 2 - Creating a new obj
[9:9:18,2010] [318] Sending data to thread
[9:9:18,2010] [0] 1 - Creating a new obj
[9:9:19,2010] [0] Memory usage at end thread 4: 214.355968MB (obj cache: 136)
[9:9:19,2010] [0] Memory usage at idle thread 4: 214.355968MB (obj cache: 136)
[9:9:19,2010] [0] Memory usage at end thread 1: 214.355968MB (obj cache: 136)
[9:9:19,2010] [0] Memory usage at idle thread 1: 214.355968MB (obj cache: 136)
[9:9:20,2010] [0] Memory usage at end thread 3: 214.482944MB (obj cache: 136)
[9:9:20,2010] [0] Memory usage at idle thread 3: 214.482944MB (obj cache: 136)
[9:9:20,2010] [0] Memory usage at end thread 2: 214.482944MB (obj cache: 136)
[9:9:20,2010] [0] Memory usage at idle thread 2: 214.482944MB (obj cache: 136)

Update (8 / 12 / 2010): just run a new Perl 5.12 compiled version of the thread and system malloc Strangely, I got the same behavior Lose a lot of MB at a time Maybe try Valgrind to see why I lost it Although I'm thinking about something else, I'm playing with something else My script creates and destroys (allegedly) many SSL sockets Is it possible for widely used modules such as IO:: socket:: SSL to leak a little? Or OpenSSL? (use v0.9.8o) To try to access the SSL module synchronously to see if there is any impact, thread access may encounter problems

Update: try to load modules separately in each thread for faster memory use Try to use the socket function to lock regions, so only one thread uses them at a time, still the same as before Increase the number of working threads from 4 to 10, and the workload is exactly the same The memory does not last for 30 minutes It leads me to believe that it is a Perl problem or stack problem (not a pun) of internal thread implementation I tried to change the stack size using the built - in threading method, but the result was the same To find another way Maybe it's a lower way Increasing the number of threads makes memory faster... Seems to be something of thread stack implementation or stack size

Update (9 / 15 / 2010): find this interesting tidbit in the IO:: socket:: SSL document

"Circular" huh? Another possible explanation is that these sockets have persisted for some time, although I explicitly define them as undef'd Go and see weaken to see if this has anything to do with sockets If I find it interesting, I'll let you know

Solved (9 / 16 / 2010): see the answer I released with this solution

Solution

Your Perl is four and a half years old Upgrade to 5.12 Just search the notes in version 5.12 and see a lot of major thread improvements. It may just magically fix your blur problem:

> 5.12 :: @_ And$_ No more thread leakage (RT #342 and #41138, also #70602, #70974) '> 5.10:: under ithreads, pl_ reg_ Regular expressions in curpm are now referenced and counted This eliminates many malicious solutions to deal with it instead of being referenced. > 5.9:: threads: several fixes, such as join () problem and memory leak In some platforms using glibc (such as Linux), the minimum memory footprint of an ithread has been reduced by several hundred kilobytes. > 5.9:: threads:: shared many memory leaks have been fixed

I mean, when you talk about four years of development and all kinds of things that may lead to this problem, look at the current threads:: shared update day

I commented on your post. This will be my next set of suggestions: if you don't use glibc and use Perl malloc (default), you will never free memory to the operating system The process size will represent the maximum size occupied by Perl at a time Try rebuilding with glibc malloc (need to recompile) to see if different memory configuration files are provided In addition, it will be time to display the code