I've created a ListView populated by the data returned from a query.
It works, but in the LogCat I've got the message:
Cursor Window: Window is full: requested allocation 444 bytes, free space 363 bytes, window size 2097152 bytes
and it uses a couple of minutes for loading / visualizing the ListView.
My query returns about 3700 rows of String/Int/Double, each of which with 30 columns; no images or particular datatypes
What does this message exactly mean and how can I avoid it?
Can you improve performances by changing this Cursor Window?
From my experience this means that the query results are too large for the cursor's window and it requests more memory. Most times this request is honored, but on low end devices it could throw exceptions.
I don't know the specifics of the app in question but you referred to a ListView. A ListView cannot show 3700 rows at once and a endless list could help to load the data on demand
My advise is to break up the query into a multiple queries that return smaller results and close them before running the next query. After each successive query combine the results.
Short version:
After some investigation, it appears that this message is part of normal operation, and not a cause for concern. It is logged at the "Warning" level, but I think this is simply overeager.
Longer version:
This is (clearly labelled as) a "Windowed" cursor, which means that old records will be discarded as new records are obtained. In the simplest form, such a "window" implementation may contain up to N rows total, possibly with some read-ahead. In this implementation, however, the window size is defined instead by the total size. The number of rows kept in memory is instead based on how many would fit in the overall window, and will vary at runtime (This could perhaps be considered more of a "buffered" Cursor than "windowed" Cursor).
As a buffered implementation with a (soft-?)capped size, the earliest rows will be discarded only when the buffer is too full to accommodate the next row. In this case, 1 or more older rows are dropped. This "keep allocating rows as-needed until we can no longer have room for more, at which point we free up the oldest record(s) in our buffer and try again" process appears to be completely normal and expected, as a normal part of the process to keep the memory space confined.
I based this conclusion on reading the source here, combined with some inference:
https://android.googlesource.com/platform/frameworks/base/+/master/libs/androidfw/CursorWindow.cpp
Why are people talking about images and other massive LOBs?
If the size of a single row is larger than the entire "window" (buffer), then this strategy breaks down and you have an actual problem.
This was the message #op was getting:
Cursor Window: Window is full: requested allocation 444 bytes, free space 363 bytes, window size 2097152 bytes
This was the message #vovahost was getting:
CursorWindow: Window is full: requested allocation 2202504 bytes, free space 2076560 bytes, window size 2097152 bytes
In the first case, requested allocation is much smaller than the windows size. I expect that similar messages are issued repeatedly, with the same window size and varying requested allocation sizes. Each time this is printed, memory is freed from the larger window, and new allocations are made. This is normal and healthy operation.
In the second case, requested allocation size exceeds the overall window size. This is an actual problem, requiring storing and reading data in a more streamable way.
The difference is "length" (total number of rows) vs "width" (memory cost of the largest single row). The former (#tirrel's issue) is not an issue, but the latter (#vovahost's issue) is.
I also got this problem. In my case I saved a 2.2 MB image in database. When loading the data from the database using Cursor.getBlob() I would see this message in the Log:
CursorWindow: Window is full: requested allocation 2202504 bytes, free space 2076560 bytes, window size 2097152 bytes
After I would get this message if I try to retrieve any data (String, number, etc) for successive rows it is returned as null without any errors.
The solution was to remove the 2.2 MB blob. I don't know if it's possible to load bigger blobs from database in Android.
Also, note that changing the window has overhead of IPC.
So, if the cursor has large number of items and is used with a listview, fast navigation results in change of window and hence frequent IPCs. This might result in ANR if the system is loaded.
Related
I am trying to store and retrieve the image captured by device camera in SQLite Database as BLOB . I have no problem to do it, but when I retrieve images from DB I get an error, because of cursor, right?
W/CursorWindow: Window is full: requested allocation 707903 bytes, free space 680839 bytes, window size 2097152 bytes
After long time looking for solution I couldn't find it. I found only a lot of questions about the same issue without solid solution.
I can store URI of image , like it is discussed Android: Cursor Window is full. But then what happens if user deletes the image from Phone Memory/SD Card?
Also its not a good idea to ignore it, answered here https://stackoverflow.com/a/37035510/8258166.
So, what should I do?
Many thanks in advance!
When I often go back and forward in my learning-app (everytime loading the question and answer) i will get this error:
01-01 15:27:36.803: E/CursorWindow(3820): Could not allocate CursorWindow '/data/data/*package-name*/databases/quiz.db' of size 2097152 due to error -12.
I can't say exactly when I get this error, only that when im often changing the content
Can anybody help me to understand what this error means, please
Thanks
According to this question and answers you should check if you handle your cursors correctly and if the amount of data you query is too big (check the 2048k limit)
I am asking this question because some answers on StackOverflow are not the ones I am looking for.
My issue is that in early versions of Android like 2.3, the query is very slow and I get this message.
My Error Message
06:37:25.521: ERROR/CursorWindow(322): need to grow: mSize = 1048576, size = 45, freeSpace() = 43, numRows = 8928
07-12 06:37:25.521: ERROR/CursorWindow(322): not growing since there are already 8928 row(s), max size 1048576This message does not happen on Android 3.0+ for me I think, but it still takes a long time to do the query, so I am looking into using limits. It works almost flawlessly on Android 4.0+ though without limits. I am using SQLite FTS3 MATCH queries as well.
The reason I have 8928 rows is because I'm doing search suggestions and someone can input a "t" and get 8928 rows or something even larger, which freezes up Android 2.3 apparently. I cannot limit my search suggestion threshold because some results with only two characters may have 20 results or 20000+ depending on what it is.
Android SQLite and huge data sets This link is probably the best place for answers. The last answer in the given link seems promising, but there is no way to get the number of rows unless you do cursor.getCount(), which requires you to get the total number of rows from a big query (ex. 20000+ results), unless I'm thinking about it wrong. Is there some workaround to this? How would you do this in code to solve the CursorWindow problem?
I have a Lucene Index with 50571 documents in it from 1740 books. I have two processes that create this index. The first process is to create the index on the device document by document. This process is very slow. The other process is to create a book index on the server, (The exact same way I create it on the device) and download and merge it with the master index. This one is much quicker to create the master index. Creating the index works fine either way.
The problem is when I search on the download-merge index I get an OutOfMemoryException, but when I search with the index that was created on the device I don't get that error. I went through and created the index book by book (download-merge) and searched after each book was indexed; based on that and when I get to book ~450 I start getting the OutOfMemoryException.
What is causing me to run out of memory.
Lucene is a memory hog. When writing "merging" indices together it stores the entire set of indices in memory twice. As quoted from the lucene documentation.
Note that this requires temporary free space in the Directory up to 2X
the sum of all input indexes (including the starting index).
If readers/searchers are open against the starting index, then temporary free
space required will be higher by the size of the starting index
That is a lot of memory. To mitigate this we have to shrink the size of the index by calling forceMerge(int) on the index writer. It is a slow process but it does shrink the size of the index. I am call this with an argument of 1 every time there is 50 or more files in the index directory.
I recently created a program that gets medi-large amounts of xml data and converts it into arrays of Strings, then displays the data.
The program works great, but it freezes when it is making the arrays (for around 16 seconds depending on the size).
Is there any way I can optimize my program (Alternatives to string arrays etc.)
3 optimizations that should help:
Threading
If the program freezes it most likely means that you're not using a separate thread to process the large XML file. This means that your app has to wait until this task finishes to respond again.
Instead, create a new thread to process the XML and notify the main thread via a Handler when it's done, or use AsyncTask. This is explained in more detail here.
Data storage
Additionally, a local SQLite database might be more appropriate to store large amounts of data, specially if you don't have to show it all at once. This can be achieved with cursors that are provided by the platform.
Configuration changes
Finally, make sure that your data doesn't have to be reconstructed when a configuration change occurs (such as an orientation change). A persistent SQLite database can help with that, and also these methods.
You can use SAX to process the stream of XML, rather than trying to parse the whole file and generating a DOM in memory.
If you find that you really are using too much memory, and you have a reason to keep the string in memory rather than caching them on disk, there are certainly ways you can reduce the memory requirements. It's a sad fact that Java strings use a lot of space. They require two objects (the string itself and an underlying char array) and use two bytes per char. If your data is mostly 7-bit ASCII, you may be better of leaving it as a UTF-8 encoded byte stream, using 1 byte per character in the typical case.
A very effective scheme is to maintain an array of 32k byte buffers, and append the UTF-8 representation of each new string onto the first empty space in one of those arrays. Your reference to the string becomes a simple integer: PTR = (buffer index * 32k) + (buffer offset). "PTR/32k" yields the index of the desired byte buffer, and "PTR % 32k" yields the location within the buffer. Use either an initial length byte or a null terminator to keep track of how long the string is. When you need to access one of the strings, don't allocate a new String object: unpack it into a mutable StringBuilder or work directly with the UTF-8 byte representation.
The above approach is obviously a lot more work, but can save you between a factor of 2 and 6 in memory usage (depending on the length of your strings). However, you should beware of premature optimization. If your problem is with the processing time to parse your input, or is somewhere else in your program, you could find that you've done a lot of work to fix something that isn't your bottleneck and thus get no improvement at all.