My app uses a proprietary implementation of Canny edge detection based on RenderScript. I tested this on numerous devices with various APIs and it worked very reliably. Now I got the new Samsung S7 working on API23. Here (and only here) I encountered a rather ugly problem. Some of the edge pictures are studded with thousands of artifacts that stem from the Magnitude gradient calculation kernel and are NOT based on actual image information. After trying with all kind of TargetAPIs, taking renderscript.support.mode on and off, etc. I finally found that the problem only arises, when the RenderScript (and Script) instances are used for the second or more times. It does not arise when using them for the first time.
For efficiency reasons I created the RenderScript and Script instances only once in the onCreate method of MainActivity and used it repeatedly thereafter. Of course I don't like to change that.
Does anyone have a solution of this problem? Thanks.
UPDATE: Crazy things are going on here. It seems that freshly created Allocations are NOT empty from the outset! When running:
Type.Builder typeUCHAR1 = new Type.Builder(rs, Element.U8(rs));
typeUCHAR1.setX(width).setY(height);
Allocation alloc = Allocation.createTyped(rs, typeUCHAR1.create());
byte se[] = new byte[width*height];
alloc.copyTo(se);
for (int i=0;i<width*height;i++){
if (se[i]!=0){
Log.e("content: ", String.valueOf(se[i]));
}
}
... the byte Array se is full of funny numbers.... HELP! Any idea, what is going on here?
UPDATE2: I stumbled over my own ignorance here - and really don't deserve a point for this masterpiece.... However, to my defense I must say that the problem was slightly more subtle that it appears here. The context was, that I needed to assign a global allocation (Byte/U8) which initially should be empty (i.e. zero) and then, within the kernel getting partially set to 1 (only where the edges are) via rsSetElementAt_uchar(). Since this worked for many months, I was not aware anymore of the fact, that I didn't explicitely assign the zeros in this allocation.... This only had consequences in API 23, so maybe this can help others not to fall into this trap.... So, note: other than numerical Arrays that are filled with 0 (as by Java default), Allocations cannot assumed to be full of zeros at initiation. Thanks, Sakridge.
Allocation data for primitive types (non-struct/object) is not initialized by default when an Allocation is created unless passed a bitmap using the createFromBitmap api. If you are expecting this then possibly you have a bug in your app which is not exposed when the driver initializes to 0s. It would help if you can post example code which reproduces the problem.
Initialize your allocations by copying from a bitmap or Java array.
I have the following code.
#include <iostream>
int * foo()
{
int a = 5;
return &a;
}
int main()
{
int* p = foo();
std::cout << *p;
*p = 8;
std::cout << *p;
}
And the code is just running with no runtime exceptions!
The output was 58
How can it be? Isn't the memory of a local variable inaccessible outside its function?
How can it be? Isn't the memory of a local variable inaccessible outside its function?
You rent a hotel room. You put a book in the top drawer of the bedside table and go to sleep. You check out the next morning, but "forget" to give back your key. You steal the key!
A week later, you return to the hotel, do not check in, sneak into your old room with your stolen key, and look in the drawer. Your book is still there. Astonishing!
How can that be? Aren't the contents of a hotel room drawer inaccessible if you haven't rented the room?
Well, obviously that scenario can happen in the real world no problem. There is no mysterious force that causes your book to disappear when you are no longer authorized to be in the room. Nor is there a mysterious force that prevents you from entering a room with a stolen key.
The hotel management is not required to remove your book. You didn't make a contract with them that said that if you leave stuff behind, they'll shred it for you. If you illegally re-enter your room with a stolen key to get it back, the hotel security staff is not required to catch you sneaking in. You didn't make a contract with them that said "if I try to sneak back into my room later, you are required to stop me." Rather, you signed a contract with them that said "I promise not to sneak back into my room later", a contract which you broke.
In this situation anything can happen. The book can be there—you got lucky. Someone else's book can be there and yours could be in the hotel's furnace. Someone could be there right when you come in, tearing your book to pieces. The hotel could have removed the table and book entirely and replaced it with a wardrobe. The entire hotel could be just about to be torn down and replaced with a football stadium, and you are going to die in an explosion while you are sneaking around.
You don't know what is going to happen; when you checked out of the hotel and stole a key to illegally use later, you gave up the right to live in a predictable, safe world because you chose to break the rules of the system.
C++ is not a safe language. It will cheerfully allow you to break the rules of the system. If you try to do something illegal and foolish like going back into a room you're not authorized to be in and rummaging through a desk that might not even be there anymore, C++ is not going to stop you. Safer languages than C++ solve this problem by restricting your power—by having much stricter control over keys, for example.
UPDATE
Holy goodness, this answer is getting a lot of attention. (I'm not sure why—I considered it to be just a "fun" little analogy, but whatever.)
I thought it might be germane to update this a bit with a few more technical thoughts.
Compilers are in the business of generating code which manages the storage of the data manipulated by that program. There are lots of different ways of generating code to manage memory, but over time two basic techniques have become entrenched.
The first is to have some sort of "long lived" storage area where the "lifetime" of each byte in the storage—that is, the period of time when it is validly associated with some program variable—cannot be easily predicted ahead of time. The compiler generates calls into a "heap manager" that knows how to dynamically allocate storage when it is needed and reclaim it when it is no longer needed.
The second method is to have a “short-lived” storage area where the lifetime of each byte is well known. Here, the lifetimes follow a “nesting” pattern. The longest-lived of these short-lived variables will be allocated before any other short-lived variables, and will be freed last. Shorter-lived variables will be allocated after the longest-lived ones, and will be freed before them. The lifetime of these shorter-lived variables is “nested” within the lifetime of longer-lived ones.
Local variables follow the latter pattern; when a method is entered, its local variables come alive. When that method calls another method, the new method's local variables come alive. They'll be dead before the first method's local variables are dead. The relative order of the beginnings and endings of lifetimes of storages associated with local variables can be worked out ahead of time.
For this reason, local variables are usually generated as storage on a "stack" data structure, because a stack has the property that the first thing pushed on it is going to be the last thing popped off.
It's like the hotel decides to only rent out rooms sequentially, and you can't check out until everyone with a room number higher than you has checked out.
So let's think about the stack. In many operating systems you get one stack per thread and the stack is allocated to be a certain fixed size. When you call a method, stuff is pushed onto the stack. If you then pass a pointer to the stack back out of your method, as the original poster does here, that's just a pointer to the middle of some entirely valid million-byte memory block. In our analogy, you check out of the hotel; when you do, you just checked out of the highest-numbered occupied room. If no one else checks in after you, and you go back to your room illegally, all your stuff is guaranteed to still be there in this particular hotel.
We use stacks for temporary stores because they are really cheap and easy. An implementation of C++ is not required to use a stack for storage of locals; it could use the heap. It doesn't, because that would make the program slower.
An implementation of C++ is not required to leave the garbage you left on the stack untouched so that you can come back for it later illegally; it is perfectly legal for the compiler to generate code that turns back to zero everything in the "room" that you just vacated. It doesn't because again, that would be expensive.
An implementation of C++ is not required to ensure that when the stack logically shrinks, the addresses that used to be valid are still mapped into memory. The implementation is allowed to tell the operating system "we're done using this page of stack now. Until I say otherwise, issue an exception that destroys the process if anyone touches the previously-valid stack page". Again, implementations do not actually do that because it is slow and unnecessary.
Instead, implementations let you make mistakes and get away with it. Most of the time. Until one day something truly awful goes wrong and the process explodes.
This is problematic. There are a lot of rules and it is very easy to break them accidentally. I certainly have many times. And worse, the problem often only surfaces when memory is detected to be corrupt billions of nanoseconds after the corruption happened, when it is very hard to figure out who messed it up.
More memory-safe languages solve this problem by restricting your power. In "normal" C# there simply is no way to take the address of a local and return it or store it for later. You can take the address of a local, but the language is cleverly designed so that it is impossible to use it after the lifetime of the local ends. In order to take the address of a local and pass it back, you have to put the compiler in a special "unsafe" mode, and put the word "unsafe" in your program, to call attention to the fact that you are probably doing something dangerous that could be breaking the rules.
For further reading:
What if C# did allow returning references? Coincidentally that is the subject of today's blog post:
Ref returns and ref locals
Why do we use stacks to manage memory? Are value types in C# always stored on the stack? How does virtual memory work? And many more topics in how the C# memory manager works. Many of these articles are also germane to C++ programmers:
Memory management
You're are simply reading and writing to memory that used to be the address of a. Now that you're outside of foo, it's just a pointer to some random memory area. It just so happens that in your example, that memory area does exist and nothing else is using it at the moment.
You don't break anything by continuing to use it, and nothing else has overwritten it yet. Therefore, the 5 is still there. In a real program, that memory would be reused almost immediately and you'd break something by doing this (though the symptoms may not appear until much later!).
When you return from foo, you tell the OS that you're no longer using that memory and it can be reassigned to something else. If you're lucky and it never does get reassigned, and the OS doesn't catch you using it again, then you'll get away with the lie. Chances are though you'll end up writing over whatever else ends up with that address.
Now if you're wondering why the compiler doesn't complain, it's probably because foo got eliminated by optimization. It usually will warn you about this sort of thing. C assumes you know what you're doing though, and technically you haven't violated scope here (there's no reference to a itself outside of foo), only memory access rules, which only triggers a warning rather than an error.
In short: this won't usually work, but sometimes will by chance.
Because the storage space wasn't stomped on just yet. Don't count on that behavior.
A little addition to all the answers:
If you do something like this:
#include <stdio.h>
#include <stdlib.h>
int * foo(){
int a = 5;
return &a;
}
void boo(){
int a = 7;
}
int main(){
int * p = foo();
boo();
printf("%d\n", *p);
}
The output probably will be: 7
That is because after returning from foo() the stack is freed and then reused by boo().
If you disassemble the executable, you will see it clearly.
In C++, you can access any address, but it doesn't mean you should. The address you are accessing is no longer valid. It works because nothing else scrambled the memory after foo returned, but it could crash under many circumstances. Try analyzing your program with Valgrind, or even just compiling it optimized, and see...
You never throw a C++ exception by accessing invalid memory. You are just giving an example of the general idea of referencing an arbitrary memory location. I could do the same like this:
unsigned int q = 123456;
*(double*)(q) = 1.2;
Here I am simply treating 123456 as the address of a double and write to it. Any number of things could happen:
q might in fact genuinely be a valid address of a double, e.g. double p; q = &p;.
q might point somewhere inside allocated memory and I just overwrite 8 bytes in there.
q points outside allocated memory and the operating system's memory manager sends a segmentation fault signal to my program, causing the runtime to terminate it.
You win the lottery.
The way you set it up it is a bit more reasonable that the returned address points into a valid area of memory, as it will probably just be a little further down the stack, but it is still an invalid location that you cannot access in a deterministic fashion.
Nobody will automatically check the semantic validity of memory addresses like that for you during normal program execution. However, a memory debugger such as Valgrind will happily do this, so you should run your program through it and witness the errors.
Did you compile your program with the optimiser enabled? The foo() function is quite simple and might have been inlined or replaced in the resulting code.
But I agree with Mark B that the resulting behavior is undefined.
Your problem has nothing to do with scope. In the code you show, the function main does not see the names in the function foo, so you can't access a in foo directly with this name outside foo.
The problem you are having is why the program doesn't signal an error when referencing illegal memory. This is because C++ standards does not specify a very clear boundary between illegal memory and legal memory. Referencing something in popped out stack sometimes causes error and sometimes not. It depends. Don't count on this behavior. Assume it will always result in error when you program, but assume it will never signal error when you debug.
Pay attention to all warnings. Do not only solve errors.
GCC shows this warning:
warning: address of local variable 'a' returned
This is the power of C++. You should care about memory. With the -Werror flag, this warning became an error and now you have to debug it.
It works because the stack has not been altered (yet) since a was put there.
Call a few other functions (which are also calling other functions) before accessing a again and you will probably not be so lucky anymore... ;-)
You are just returning a memory address. It's allowed, but it's probably an error.
Yes, if you try to dereference that memory address you will have undefined behavior.
int * ref () {
int tmp = 100;
return &tmp;
}
int main () {
int * a = ref();
// Up until this point there is defined results
// You can even print the address returned
// but yes probably a bug
cout << *a << endl;//Undefined results
}
This behavior is undefined, as Alex pointed out. In fact, most compilers will warn against doing this, because it's an easy way to get crashes.
For an example of the kind of spooky behavior you are likely to get, try this sample:
int *a()
{
int x = 5;
return &x;
}
void b( int *c )
{
int y = 29;
*c = 123;
cout << "y=" << y << endl;
}
int main()
{
b( a() );
return 0;
}
This prints out "y=123", but your results may vary (really!). Your pointer is clobbering other, unrelated local variables.
That's classic undefined behaviour that's been discussed here not two days ago -- search around the site for a bit. In a nutshell, you were lucky, but anything could have happened and your code is making invalid access to memory.
You actually invoked undefined behaviour.
Returning the address of a temporary works, but as temporaries are destroyed at the end of a function the results of accessing them will be undefined.
So you did not modify a but rather the memory location where a once was. This difference is very similar to the difference between crashing and not crashing.
In typical compiler implementations, you can think of the code as "print out the value of the memory block with adress that used to be occupied by a". Also, if you add a new function invocation to a function that constains a local int it's a good chance that the value of a (or the memory address that a used to point to) changes. This happens because the stack will be overwritten with a new frame containing different data.
However, this is undefined behaviour and you should not rely on it to work!
It can, because a is a variable allocated temporarily for the lifetime of its scope (foo function). After you return from foo the memory is free and can be overwritten.
What you're doing is described as undefined behavior. The result cannot be predicted.
The things with correct (?) console output can change dramatically if you use ::printf but not cout.
You can play around with debugger within below code (tested on x86, 32-bit, Visual Studio):
char* foo()
{
char buf[10];
::strcpy(buf, "TEST");
return buf;
}
int main()
{
char* s = foo(); // Place breakpoint and the check 's' variable here
::printf("%s\n", s);
}
It's the 'dirty' way of using memory addresses. When you return an address (pointer) you don't know whether it belongs to local scope of a function. It's just an address.
Now that you invoked the 'foo' function, that address (memory location) of 'a' was already allocated there in the (safely, for now at least) addressable memory of your application (process).
After the 'foo' function returned, the address of 'a' can be considered 'dirty', but it's there, not cleaned up, nor disturbed/modified by expressions in other part of program (in this specific case at least).
A C/C++ compiler doesn't stop you from such 'dirty' access (it might warn you though, if you care). You can safely use (update) any memory location that is in the data segment of your program instance (process) unless you protect the address by some means.
After returning from a function, all identifiers are destroyed instead of kept values in a memory location and we can not locate the values without having an identifier. But that location still contains the value stored by previous function.
So, here function foo() is returning the address of a and a is destroyed after returning its address. And you can access the modified value through that returned address.
Let me take a real world example:
Suppose a man hides money at a location and tells you the location. After some time, the man who had told you the money location dies. But still you have the access of that hidden money.
Your code is very risky. You are creating a local variable (which is considered destroyed after function ends) and you return the address of memory of that variable after it is destroyed.
That means the memory address could be valid or not, and your code will be vulnerable to possible memory address issues (for example, a segmentation fault).
This means that you are doing a very bad thing, because you are passing a memory address to a pointer which is not trustable at all.
Consider this example, instead, and test it:
int * foo()
{
int *x = new int;
*x = 5;
return x;
}
int main()
{
int* p = foo();
std::cout << *p << "\n"; // Better to put a newline in the output, IMO
*p = 8;
std::cout << *p;
delete p;
return 0;
}
Unlike your example, with this example you are:
allocating memory for an int into a local function
that memory address is still valid also when function expires (it is not deleted by anyone)
the memory address is trustable (that memory block is not considered free, so it will be not overridden until it is deleted)
the memory address should be deleted when not used. (see the delete at the end of the program)
I'd like to ask for some help about the following problem I have.
I'd like to create an application that solves the Rubik's cube with an optimal solution. I downloaded the following library, whitch supposedly does just that using the Kociemba's Algorithm.
http://kociemba.org/twophase.jar
Apparently it can solve the cube in under 0.5 sec, but in my app it never returned the solution due to memory problems. I know it works, I tested it with wrong inputs and it returns the documented error codes.
I call it in my onCreate method like this:
resultTxt = Search.solution(data, 21, 10, true);
resultTxt is a String variable and it should contain the solution.
It quickly eats up the memory.
I tried it with IntentService without success. By this I mean it didn't really changed anything.
As i didn't find any evidence of anyone using this library in any android application, I thought I would ask someone who is more experienced than me.
Is there any way I could make this work on Android, or is this as impossible as I thought?
It may be a bit late, but I was also facing this issue quite recently when I was working on a Rubik's-Cube-solving-robot using an Android-Smartphone for scanning the cube and computing the solution, so I'll put here what I have found out.
What is the problem?
Let's start off by discussing where the problem causing that performance issue actually is located.
The reason for that being so slow is the class CoordCube, which looks (very simplified) like this:
class CoordCube {
short[] pruneTables;
static {
/* compute and save ~50MB data in `pruneTables` */
}
}
Basically what it does, is to load a pretty big amount of data into lookup-tables which are required for a fast solving procedure. This loading is automatically executed by the JVM once this class is first instantiated. That happens on line 159 in Search.solution():
/* simplified code of solution() */
if (cube.isValid()) {
CoordCube c = new CoordCube(); // pruning tables will be loaded on this line
That is also the reason why this method executes in negligible time as long as the passed cube is invalid, since it never gets to load the tables.
Possible Solutions:
Now that we have identified where the problem is located, let's focus on how to solve it.
I have come up with 3 different approaches, where the first one is probably the easiest (but also the slowest execution wise ...) and is also used in my App. The other two are just ideas on how to improve the performance even more.
Approach 1:
The first and most simple approach is to just manually preload the lookup tables in a kind of LoadingActivity with a ProgressBar showing our current progress. For that we first want to be able to manually control exactly when which tables are loaded (when the class is first instantiated is not precise enough), like this:
loadTable1() {
/* load pruning table number 1 */
}
For that I have written some simple utility here (code is too long to paste). Make sure to check out my instructions there on how to properly import that code in your App.
Also we will probably want to do the loading in the background, namely in an AsyncTask. This is how I have done it in my application (PruneTableLoader is included in the previous link):
private class LoadPruningTablesTask extends AsyncTask<Void, Void, Void> {
private PruneTableLoader tableLoader = new PruneTableLoader();
#Override
protected Void doInBackground(Void... params) {
/* load all tables if they are not already in RAM */
while (!tableLoader.loadingFinished()) { // while tables are left to load
tableLoader.loadNext(); // load next pruning table
publishProgress(); // increment `ProgressBar` by one
}
return null;
}
#Override
protected void onProgressUpdate(Void... values) {
super.onProgressUpdate(values);
/* increment `ProgressBar` by 1 */
}
}
When using my PruneTableLoader, the loading of all tables needs about 40s on my Samsung Galaxy S3 with 250 MB RAM free. (in contrast it needs well over 2min when loading them automatically and in addition often results in a crash ...)
That may still sound quite slow considering it needs < 1s on PC, but at least you must only do that once, since Android caches the static-variables and you so don't have to load them on every startup of your App.
Approach 2: (untested)
I assume it would be faster to save the pruning tables in a file or a database and load them from there instead of always recomputing them. I have not yet tested that though and it will probably require quite some work getting the saving and loading to work properly. (also maybe it's not even faster because of access times)
Approach 3: (untested)
Well, the hardest and also by decades most work expensive solution would be, to simply rewrite the whole algorithm in C or C++ and invoke it in the App via JNI. (Herbert Kociemba has not published his C-sourcecode yet as far as I know ...)
This is going to be the performance wise fastest solution for sure. (also for the solving procedure itself)
All in all approach 1 is probably the effort/benefit-wise best approach for the beginning (and also was for me), so I would recommend you to go with that, in case the loading time is not such a huge issue for your Application.
I'm not completely satisfied with the performance of that myself though, so I may try out approach 2 and maybe even approach 3 some time in the future. In case I do that, I will update this post with my results then.
I experience some memory leaks in my android application. I've already used MAT to analyze the memory usage. But I have one question from the DDMS perspectiv in Eclipse, what does 1-byte array[byte[], boolean[]) mean?
Is this specific for my application? You can see that this is the big memory leak issue, this always increases in size, and the rest will increase and decrease randomly. My guess is that the GC doesn't catch this type. Can anybody explain why this happen, with this little information?
One byte array is the designation for any data structure that is organized as a single byte array. In you case and with that size, I would bet in a Bitmap or a Drawble.
Most common reasons for memory leaks are static object not properly managed and holding references to:
Context
View (which holds reference to context (and possibly also to bitmap)
Thread (which are not easly collected by GC)
Handler (which holds reference to context)
Most of them can be solved ensuring that you set the object to null when it's no long required.
Regards.
A byte and a boolean are each 1 byte. If you have an array of those you have a "1-byte array".
A ByteBuffer for example should internally hold one of those.
You have a total of 614 of them where the smallest one be a byte[24] (or boolean[24]), the largest one is 3MB. All of them together use 104MB.
The GC will get rid of them if they are no longer referenced.
For example when you put
private static byte myArray[] = new byte[3 * 1024 * 1024];
in one of your classes and never set it back to null (myArray = null) then this one can't be garbage collected because another Object has a reference to it. The object would be the class itself (as in String.class). The class object can't be garbage collected since classes are never unloaded. (they are referenced by their ClassLoader which could itself be referenced / loaded by another ClassLoader and you can't create Objects & use classes without them - they need to stay and they need to keep a reference to their classes)
It's usually not that simple but often start with static somewhere.
Within MAT (after you force GC) look at the reference chain for the objects that are no longer intended to stay alive and identify the one Object that holds the reference. Set that one to null in your code and your leak is gone. Explained in more detail here:
http://android-developers.blogspot.de/2011/03/memory-analysis-for-android.html
I ran to this problem tonight and almost I checked every bit of code but I couldn't find anything.
What I did was starting the app from intelij and then pressing home button and open the app again. Every time the app heap was doubled.
Finally I discover when I launch the app from ADB and press the home button and open the app again it doesn't bring back the old activity, it just start a new one. When I pressed finish it starts to cycle through all of them. It's like they are treated as two different intent. So I put android:launchMode="singleTop" on the main activity in manifest and it resolved the problem.
Although it's not the main cause of this problem but if you encountered this check this out before anything. It wasted three or four hours for me.
I tried to make a .map file to use it in my android application. My point is to show map on screen with help of mapsforge library. Library requires a .map file. The problem is generating it with osmosis mapfilewriter plugin. When I use command
./osmosis --rx file=/home/user/poland.osm --mw file=result.map bbox=51.09,16.9,51.14,17
I get
SEVERE: Thread for task 1-rx failed
java.lang.OutOfMemoryError: Java heap space
I tried using another order of arguments but it also doesn't work. I also tried using -Xmx1200m option to increase javas heap memory but usage of memory was this same and didn't help.
I would be grateful for help.
Use the parameter type=hd.
The type parameter has two options ram and hd. Although I don't see that the documentation doesn't explicitly state what hd means, I would guess it means "hard disk", meaning it would offload data to disk, rather than keeping it all in memory.
The resulting command would be then:
./osmosis --rx file=/home/user/poland.osm --mw file=result.map bbox=51.09,16.9,51.14,17 type=hd
If you have enough memory, you need to tell Java about it (it will only use a limited amount otherwise); but this is not an argument passed directly to osmosis.
On Windows, you can follow this advice given by Emilie Laffray on the OSM-dev list:
In osmosis.bat is the following line... REM # JAVACMD_OPTIONS - The
options to append to the java command, typically used to modify jvm
settings such as max memory.
so, either modify osmosis.bat or, create a new file called osmosis.bat
in the all users profile directory or your profile directory, to
include a 'set JAVACMD_OPTIONS = -Xmx1024M' line (to set maximum
memory usage to 1GB, if you have enough RAM)...
On Linux (and I'd assume Mac OS X also), you can create the file ~/.osmosis
containing the line
JAVACMD_OPTIONS=-Xmx1024M
which will be loaded by osmosis on startup. Note that both examples above use the value 1024M, i.e. you're giving Java access to 1024 MB of RAM - you could use a different value depending on your system.