Java – how can pattern search be faster?

I'm processing about 1GB of incremental files and I want to search for a specific pattern

Solution

Basically, what you need is a state machine that can handle streams The stream is bound to the file... Each time the file grows, you will read the attached content (such as the tail Linux command of the standard output attached to the line of the file)

If you need to stop / restart the analyzer, you can store it somewhere in the starting position (depending on the pattern matching window you need) and restart it Or you can restart from scratch

That's part of the problem of "adding files"

The best way to deal with content depends on what you really need and which data and schema to apply Regular expressions may be the best solution: flexible, fast and convenient

From my understanding, if you want to search and match some natural language content, Lucene will be very good It would be a bad choice to match all dates or all lines with a particular property It's also because Lucene first creates an indexed document... This will help only really heavy processing, because indexing takes time first