Maximize Java heap space

I try to use a very large square matrix in Java, n = 1e6 or greater Matrices are not sparse, so I don't see many ways to represent them as 2D arrays, which requires n ^ 2 * sizeof (int) bit memory Obviously, I'm getting a heap overflow error, even if I add a compiler flag to use the heap that my machine will allow

I would like to assume that I have a perfect computer (unlimited ram, etc.) for this problem, although in reality, I am on a 64 bit machine with 16 gig ram It seems that my machine is so relevant because I am limited by the JVM rather than my actual hardware (because the JVM cannot have more memory than my physical machine)

I understand (and have been quoted, for example, making a very large Java array). In theory, Java arrays cannot be compared with Max for indexing_ Int large

My question is: is there any way to trick the extra memory out of the JVM heap

I understand that if there is, they may not give me more information

for example

In C, I can declare static constant variables and move them to the data part of the code. This part will have more space than the heap, which is much larger than the stack (where are static variables stored (in C / C + +)

In Java, it seems that even if the variable is copied into the "data" section, the value will enter the main heap static allocation in Java – heap, stack and permanent generation, which means that I have successfully removed an entire byte (yay!) from the heap

My solution

My "solution" is not a real solution I made a simple data structure, using randomfileaccess IO procedure to replace array access, read and write external files It is still a continuous time access, but we move from the fastest operation in Java to a very slow process (although we can extract "cached" lines from the file at one time, which makes the process faster) Better idea?

It's not my problem

I'm not asking how to make an array larger than the maximum array size of Java It's impossible. These are nested arrays – a single array of N sizes is good, n of which cause problems

I'm not asking how to deal with "Java. Lang. outofmemoryerror: Java heap space" error (64MB heap size) Garbage collection is irrelevant - I can't even make arrays, let alone worry about when they will be deleted

Nor can I use iterators (I think), otherwise it would be a possibility; Functions like matrix multiplication need to be able to index directly

Note: Java is not the right language to operate on very large matrices I will use the abacus better But I'm here. It's beyond my control

Solution

Your original question has some missing aspects; For example, I can't believe you have to use such large matrices and just "forget them" between runs Well, maybe you do, I don't know

Anyway: your RandomAccessFile is, IMHO, almost there; Only when I am you, I will use filechannel map(). On UNIX system, it is basically a way to call MMAP (2) In the following case, I assume you have a filechannel to your matrix (I think you know what I mean)

Because you use a matrix, it looks like the values on any given "coordinate" in the matrix have the same length, which means that you can easily calculate the offset in the file to read and / or write the given value to the matrix Of course, you don't want to map the value, but a window containing the value; Make the window large enough to be useful without worrying about heap space consumption: filechannel Map () takes no space (saves object bookkeeping) On a 64 bit JVM, you don't have to worry; If you are using a 32 - bit JVM, you have to consider that the address space is exhausted

Of course, there are expiration questions: how long do you need or how long does this mapping remain active It all depends on your program and what you do But using filechannel and mapping related areas is the way to go However, it should be reminded that mapping more than 2 ^ 31 – 1 bytes is not safe; For example, determine 2 ^ 30 (1 GIB) byte windows; And remind you that you can convert bytebuffers to intbuffers

Edit: some related links:

> FileChannel. open(); > FileChannel. map(); > ByteBuffer and its asintbuffer() method; > IntBuffer.