Java – ActiveMQ oom with stomp enabled

After enabling the stomp protocol on the ActiveMQ server (before it only enables the default protocol), it starts to fail and displays oom I have only one client using stomp It can work for a week without failure, or fail a day after restart This is the configuration file:

<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

    <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
          lazy-init="false" scope="singleton"
          init-method="start" destroy-method="stop">
    </bean>

    <!--
        The <broker> element is used to configure the ActiveMQ broker.
    -->
    <broker useJmx="true" xmlns="http://activemq.apache.org/schema/core" brokerName="cms-mq" dataDirectory="${activemq.data}">

        <destinationInterceptors>
            <virtualDestinationInterceptor>
                <virtualDestinations>
                    <virtualTopic name="VirtualTopic.>" selectorAware="true"/>
                </virtualDestinations>
            </virtualDestinationInterceptor>
        </destinationInterceptors>

        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="false">
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="false">
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>

        <managementContext>
            <managementContext createConnector="false"/>
        </managementContext>
        <persistenceAdapter>
            <kahaDB directory="${activemq.data}/kahadb"/>
        </persistenceAdapter>
          <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage percentOfJvmHeap="70" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="4 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="4 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

        <transportConnectors>
            <transportConnector name="auto" uri="auto+nio://0.0.0.0:61616?maximumConnections=1000&amp;auto.protocols=default,stomp"/>
        </transportConnectors>
        <shutdownHooks>
            <bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
        </shutdownHooks>

        <plugins>
          ... security plugins config...
        </plugins>

    </broker>

    <import resource="jetty.xml"/>

</beans>

Start args:

/usr/java/default/bin/java -Xms256M -Xmx1G -Dorg.apache.activemq.UseDedicatedTaskRunner=false -XX:HeapDumpPath=/var/logs/heapDumps -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.port=8162 -Dcom.sun.management.jmxremote.rmi.port=8162 -Dcom.sun.management.jmxremote.password.file=/opt/apache-activemq-5.13.0//conf/jmx.password -Dcom.sun.management.jmxremote.access.file=/opt/apache-activemq-5.13.0//conf/jmx.access -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/apache-activemq-5.13.0//tmp -Dactivemq.classpath=/opt/apache-activemq-5.13.0//conf:/opt/apache-activemq-5.13.0//../lib/ -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Dactivemq.conf=/opt/apache-activemq-5.13.0//conf -Dactivemq.data=/opt/apache-activemq-5.13.0//data -jar /opt/activemq/bin/activemq.jar start

UPD: from eclipse memoryanalizer:

Leak SUSPECTs
247,036 instances of "org.apache.activemq.command.ActiveMQBytesMessage",loaded by "java.net.urlclassloader @ 0xc02e9470" occupy 811,943,360 (76.92%) bytes. 

81 instances of "org.apache.activemq.broker.region.cursors.FilePendingMessageCursor",loaded by "java.net.urlclassloader @ 0xc02e9470" occupy 146,604,368 (13.89%) bytes.

UPD: before the oom error occurs, there are several errors in the log, as follows:

| ERROR | Could not accept connection from null: java.lang.IllegalStateException: Timer already cancelled. | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[cms-mq] Task-13707
| INFO  | The connection to 'null' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[cms-mq] Task-13738

Will debug anything to help debug it Additional information can be provided if required

Solution

One guess is that you are getting messages from the producer through stomp and eventually destroying the agent memory You have turned off producer traffic control, and you can even do this through the default JMS client, and stomp is easier to enter this situation, because there is no ack allowed traffic control mechanism back to the producer by default. You must request a receipt every time you send it

To debug this operation, you need to start checking the agent log and target and usage statistics through the console or other tools you choose to view the status of the agent