java - Android: Convert Stream to String without running out of memory

Question

I have an android client that communicates with the server via REST-ful endpoints and JSON. Because of this, I have the need to retrieve the full server response prior to converting it into a Hash. I have this code in place to do that (found on the internet someplace):

private static String convertStreamToString(InputStream is) {

    BufferedReader reader = new BufferedReader(new InputStreamReader(is));
    StringBuilder sb = new StringBuilder();

    String line = null;
    try {
        while ((line = reader.readLine()) != null) {
            sb.append(line + "
");
        }
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        try {
            is.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
    return sb.toString();
}

The code works for the most part, however I am seeing reports of crashes in the field from clients with an OutOfMemory exception on the line:

    while ((line = reader.readLine()) != null) {

The full stack trace is:

java.lang.RuntimeException: An error occured while executing doInBackground()
    at android.os.AsyncTask$3.done(AsyncTask.java:200)
    at java.util.concurrent.FutureTask$Sync.innerSetException(FutureTask.java:273)
    at java.util.concurrent.FutureTask.setException(FutureTask.java:124)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:307)
    at java.util.concurrent.FutureTask.run(FutureTask.java:137)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:561)
    at java.lang.Thread.run(Thread.java:1102)
Caused by: java.lang.OutOfMemoryError
    at java.lang.String.(String.java:468)
    at java.lang.AbstractStringBuilder.toString(AbstractStringBuilder.java:659)
    at java.lang.StringBuilder.toString(StringBuilder.java:664)
    at java.io.BufferedReader.readLine(BufferedReader.java:448)
    at com.appspot.myapp.util.RestClient.convertStreamToString(RestClient.java:303)
    at com.appspot.myapp.util.RestClient.executeRequest(RestClient.java:281)
    at com.appspot.myapp.util.RestClient.Execute(RestClient.java:178)
    at com.appspot.myapp.$LoadProfilesTask.doInBackground(GridViewActivity.java:1178)
    at com.appspot.myapp.$LoadProfilesTask.doInBackground(GridViewActivity.java:1)
    at android.os.AsyncTask$2.call(AsyncTask.java:185)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:305)
    ... 4 more

My question: is there any way to solve this problem aside from sending smaller chunks of data from the server?

Thanks!

Answer 1

In general, the answer is no, but you can certainly tune the conditions that cause you to run out of memory. In particular, if you send string length ahead of your stream, you will be able to create a StringBuilder with correct array size inside of it. Arrays cannot be resized after creation, so if you run out of array capacity in StringBuilder, the implementation has to allocate a new array (typically twice the size to avoid too many resizes) and then copy the old array contents. Consider stream of size X, to resize StringBuilder that just happened to be a X-1 capacity, you need almost X*3 amount of memory. Sizing StringBuilder such that resizes are avoided will allow you squeeze larger streams into memory.

Another thing you may want to do is to tune the amount of memory available to your server process. Use a switch like -Xmx1024m when launching the server process.

Of course, it would be much better to revise your algorithm to not require the entire stream to be held in memory. It will enable you to handle more clients with the same amount of hardware.