Java – ColdFusion memory leak when calling CFC directly without method parameters

We had memory problems for a long time I finally found out how to copy the problem, but I don't know what caused it or how to solve it

We have many CFCs in the web accessible / controller directory for submission and processing When CFC is called directly without method parameters, the server starts chewing memory

For example, like http://www.domain.com/controller/LoginController.cfc Such a URL will run in the browser until it times out/ Cfide is locked and cannot be accessed publicly

So cfexplorer is not (or should not) available

We use fusionreactor to monitor our instances Our server is set to 20GB heap space When the application is loaded and restarted, the memory will be about 800MB

Under normal traffic, the memory will fluctuate between 5GB and 10GB, and garbage collection will be carried out regularly After a period of time, the server finally reaches 98% capacity It often runs there

Sometimes it may last for hours or even days until some traffic peak pushes it and an external error occurs Garbage collection does not recover memory and is not active

Fusionreactor reports long - running threads Only a server restart can restore it

Using fusionreactor (we just installed it, which is the last thing I know about this problem) I'm checking permgen memory space and find that it takes up

85% of the heap This seems wrong I performed a memory dump and loaded it into map through eclipse for analysis I found 10 objects in memory

Measurement 1.7GB (1.7) × 10 (about 85% of the total reactor) These objects are as follows:

Class Name |  Shallow Heap | Retained Heap | Percentage
byte[1769628928] @ 0x4d963b198  ...128.................POST......../controller/LoginController.cfc......../controller/LoginController.cfc........173.14.93.66........173.14.93.66........www.domain.com........443........HTTP/1.1.......;D:\websites\domain\system\controller\Lo...| 1,769,628,944 | 1,944 |   8.60%

So I restarted CF on one of our servers. I checked fusionreactor and didn't see the memory usage Then go to the browser and call CFC first:

http://www.domain.com/controller/LoginController.cfc?method=foo

This causes the onmissingmethod handler to correctly kick and redirect to the appropriate error page without server effect

But later called:

http://www.domain.com/controller/LoginController.cfc

Causes the page to hang Fusionreactor reports that there are no active requests, even if one of them is running, which is why we can't identify the problem when it occurs To make matters worse, refreshing memory slowly increases it by a tenth of a percentage point without reporting activity The timeout on the server is set to 5 minutes I assume it will eventually be killed and orphaned as 1.7 GB This does not slow down the server, but runs the memory at an average of 3gb without any recovery from garbage collection This seems to explain why random calls to these URLs slowly chew and remain in memory over time

Next, I called the URL from multiple browser tabs This almost instantly soared memory to 98% Although there are more than 15 browser tags running, fusionreactor now displays two long-running requests for 10 seconds and climbs Forcibly killing threads seems powerless The problem can only be solved by restarting the server

So now I have clearly identified this problem (phantom thread creates huge isolated objects in permgen heap) and how to copy it

I don't know how or why to make a request directly to CFC It may be robots or occasionally strange browser behavior

All the huge objects are jrun servlet. jrpp. An instance of proxyendpoint

What caused this problem and how should I solve it

This is running Java 1.7 0_ Cf9 on Win2003 server of 25 01 Standard.

thank you!

Solution

I believe this is a legitimate error in ColdFusion, and I have reported it through their bug system This problem can be partially repeated on other systems For example, on the MBP where I run CF on Apache, direct CFC calls will not cause memory problems, but will cause the jrun "internal server error" page to appear immediately So there are some errors and the system is dealing with the problem in different ways Anway

Thanks to @ iknowkung Foo and a lot of experiments, I found a solution

Inserting the method key / value into the URL range seems to solve the problem It should be noted that it must be completed in the onrequeststart method, not in the oncfcrequest method It can be seen from the document that the call to CFC will go directly to oncfcrequest, but this is not the case All requests first pass through the onrequeststart method When onrequeststart returns only, oncfcrequest will call and only when the required 'method' parameter exists

So in this case, oncfcrequest is never called because the 'method' parameter never exists So here is the code to run immediately in onrequeststart:

<cfif Right(arguments.targetPage,4) IS ".cfc"
      AND NOT StructKeyExists(URL,"WSDL")
      AND NOT StructKeyExists(URL,"method")
      AND NOT StructKeyExists(FORM,"method")>
    <cfset StructInsert(FORM,"method","")>
    <cfset StructInsert(URL,"")>
</cfif>

This code checks the extension on the requested page. If there are no method parameters in the URL and form ranges, a blank key / value pair will be inserted in both for measurement Check that the 'WSDL' parameter is right there, because I found that although this code works well, suddenly we have cracked several web service CFC calls If the call to CFC is WebService cfc？ WSDL, the method parameter is not required, and CF processes the whole event in different ways

Therefore, inserting an empty 'method' value will cause oncfcrequest to be called correctly when onrequsetstart completes When calling CFC with an invalid empty method name, onmissingmethod is now started correctly This method will immediately process the wrong page request and redirect to the custom error page

Since the implementation of this fix, we have seen a consistent decline in memory usage from 98% to 15% on all servers The memory chart shows the expected amount of memory being used and collected Overall performance has changed from an average page request time of 1200 milliseconds to 54 milliseconds, and not all of these requests are rampant behind the scenes

I still hope adobe can identify and solve the problem