Java – solve problems larger than memory limit

I've been thinking about the following scenario recently: suppose you have a huge database and you want to perform some calculations when loading part of it The situation may be that a small part of the database may not be suitable for Java heap memory, which is very limited How do people solve these obstacles? How does Google analyze terabyte data with limited memory space?

Thank you in advance for your reply

Solution

The short answer is that you need to process the data in blocks suitable for memory, and then combine the results of these block calculations into the final answer (which may be divided into multiple stages) A common distributed example is map reduce: for details about Google's original implementation, see here; For more information about the open source implementation, see Hadoop