Java – ignore the record of repeated exception messages from third-party libraries

I need to deal with the repetition of specific exceptions in the log

I use slf4j and logback to log in to my application I use some external services (DB, Apache Kafka, third-party libraries, etc.) When a connection is lost to such a service, I get exceptions, such as

[kafka-producer-network-thread | producer-1] WARN  o.a.kafka.common.network.Selector - Error in I/O with localhost/127.0.0.1
java.net.ConnectException: Connection refused: no further information
    at sun.nio.ch.socketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_45]
    at sun.nio.ch.socketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_45]
    at org.apache.kafka.common.network.Selector.poll(Selector.java:238) ~[kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) [kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) [kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) [kafka-clients-0.8.2.0.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

The problem is that I get this message every second This exception message will flood my log file, so I will have several GB in the log file in n hours

I want a log message about this exception every 1-5 minutes Is there any way to ignore the abnormal duplication in the log file?

Possible solutions:

>Ignore all logs for specific packages and classes [bad, because I can skip important information] > use http://logback.qos.ch/manual/filters.html#DuplicateMessageFilter [bad, because I can only set the attribute allowedrepetitions or cachesize. It will match all messages, but I only need specific exceptions] > write a custom filter. Maybe, do you know it has been solved?

Solution

I think your best choice is to extend the duplicatemessagefilter you have found It's not final, it's easy:

>Implement the new turbofilter > using the classname or exceptiontype based filtering method or anything else you want to make a preliminary decision based on it, and then delegate the parent class for duplication check

Available parameters:

public FilterReply decide(Marker marker,Logger logger,Level level,String format,Object[] params,Throwable t) {

Including throwable and logger