General architecture of data processing system running in Java for a long time?

I was asked to port legacy data processing applications to Java

The current version of the system consists of nubmer of (poorly written) excel tables The worksheet implements a big loop: polling many data sources These sources are a mixture of CSV and XML - based web services

The process is conceptually simple:

It is stateless, which means that the calculations run depend entirely on the input Publish the calculation results (at present, a large number of CSV files are written in some standard locations on the network)

After the results are published, the polling cycle starts again

This process does not require an administrator GUI, but if I can implement some kind of Web - based control panel, it will be concise For internal use, it will be very beautiful and pure The control panel only executes scheduling statistics about the source, and may force a refresh of the input source when a problem occurs This component is purely optional in the first round of delivery

A key feature of the system is fault tolerance Some input sources are well known errors I hope my system can recover when some inputs are damaged In this case, the output cannot be updated - I want to continue polling before the system resolves, and some XMPP messages may be generated to indicate the status of the system Overall, the system should work without prolonged intervention

The user currently has a custom client for polling CSV files (hopefully) without rewriting If I can do this correctly, they won't notice that the engine running the system has been re implemented

I am not a Java devloper (I mainly work in Python), but JVM is the requirement in this case The manager gave me a lot of study time

What I want to know is how to start building such a project I want to make use of frameworks and & good patterns Are there any important building blocks that can help me achieve better quality and system speed?

Update 0: no one has mentioned spring yet - can this framework work in such applications?

Solution

You can use many large and complex frameworks to "help" do this These can be resumes

In your case, I suggest you try to make the system as simple as possible It will perform better and easier to maintain (it is also more likely to work)

So I will consider every request and ask myself; Can I make it so simple? It's not laziness (you have to work harder), but good practice, with all due respect