Download files of unknown length over HTTP using java

I want to download HTTP query in Java, but the length of the file I downloaded is uncertain

I think this is very standard, so I searched and found its code fragment: http://snipplr.com/view/33805/

But it has a problem with the contentlength variable Since the length is unknown, I get - 1 back This can produce errors When I omit the entire check on contentlength, this means that I always have to use the maximum buffer

The problem is that the document is not ready yet Therefore, flush will only be partially populated and some files will be lost

If you try to use this clip to download something similar http://overpass-api.de/api/interpreter?data=area%5Bname%3D%22Hoogstade%22%5D%3B%0A%28%0A ++Node% 28area% 29% 3B% 0A + +% 3C% 3B% 0A% 29 +% 3B% 0aout + meta + QT% 3B link, you will notice this error, and when you always download the maximum buffer to omit the error, you will eventually get a corrupted XML file

Is there any way to download only the preparation part of the file? I think if this can download large files (up to a few GB)

Solution

This should work, I test it, it applies to me:

void downloadFromUrl(URL url,String localFilename) throws IOException {
    InputStream is = null;
    FileOutputStream fos = null;

    try {
        URLConnection urlConn = url.openConnection();//connect

        is = urlConn.getInputStream();               //get connection inputstream
        fos = new FileOutputStream(localFilename);   //open outputstream to local file

        byte[] buffer = new byte[4096];              //declare 4KB buffer
        int len;

        //while we have availble data,continue downloading and storing to local file
        while ((len = is.read(buffer)) > 0) {  
            fos.write(buffer,len);
        }
    } finally {
        try {
            if (is != null) {
                is.close();
            }
        } finally {
            if (fos != null) {
                fos.close();
            }
        }
    }
}

If you want it to run in the background, you only need to call it in one thread:

Thread download = new Thread(){
    public void run(){
        URL url= new URL("http://overpass-api.de/api/interpreter?data=area%5Bname%3D%22Hoogstade%22%5D%3B%0A%28%0A++node%28area%29%3B%0A++%3C%3B%0A%29+%3B%0Aout+Meta+qt%3B");
        String localFilename="mylocalfile"; //needs to be replaced with local file path
        downloadFromUrl(url,localFilename);
    }
};
download.start();//start the thread