Essentially the question was in title. In detail: I am using Realm for storing
users in device storage. I am doing one request to server which returns list with 110k objects, gson parses this, all actions from out of memory were done. On emulator this operation continues for about 3 minutes and all saved to db successfully. All json data is valid, if it wasn't, on emulator it would crashed too. But, when I checked on real android device I got this
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Expected ':' at line 2 column 3 path $[70816].info2000
when gson was trying to parse 70816 object it couldn't, because it's not valid. I re-checked this user in database. All was good. I ran again and got this exception on another object
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Expected name at line 2 column 1091 path $[56000].accountCrm
So it's randomly.
I got confused and began researching, and I couldn't find anything. These objects come in response body(okhttp), I parse it like this:
Gson gson = GsonBase.getGlobalGson();
JsonReader reader = new JsonReader(responseBody.charStream()); //these are my 110k objects
reader.setLenient(true);
reader.beginArray();
List<ContactCRMRealm> contactCRMRBuffer = new ArrayList<>();
int counter = 0;
while (reader.hasNext()) {
if (counter < BUFFER_SIZE) {
contactCRMRBuffer.add(gson.fromJson(reader, ContactCRMRealm.class));
counter++;
} else {
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
counter = 0;
contactCRMRBuffer.clear();
}
}
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
contactCRMRBuffer.clear();
reader.endArray();
Exceptions are thrown in this line:
contactCRMRBuffer.add(gson.fromJson(reader, ContactCRMRealm.class));
I can't figure out but It seems like response body is being broken unexpectedly. Why It works on emulator and doesn't on android device? Few things that I can say: when I comment this line in else block:
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
all parsing side was performed without exceptions. I think it's because of realm. But what exactly? I re-wrote this parsing using createOrUpdateAllFromJson() realm method:
defaultInstance.beginTransaction();
try {
defaultInstance.createOrUpdateAllFromJson(ContactCRMRealm.class, responseBody.byteStream());
defaultInstance.commitTransaction();
} catch (IOException e) {
defaultInstance.close();
} finally {
defaultInstance.close();
}
On emulator was out of memory(wtf), on real device nothing happened for 10 minutes and login exception was thrown(read timeout 10 minutes).Also I was trying to make offset, by 10 thousands users on one request, nothing changed. Please help.
Ok, after a day of investigation, the problem was solved. Realm was 2.3.0 version in the app(yes, really old), so I was thinking maybe need to upgrade to newer verison. I did it, I upgraded realm to 3.7.0(not the last one because rxjava1 is using in the app). That's it. There are a lot of improvements, especially speed improvements, so now all is fine.
Related
I am facing a strange issue, and I am not able to debug it out. I have implemented a logic for uploading stream of data and am using Volley for the same, I have customized a logic little bit in HurlStack, addBodyIfExists api,so that body of type "application/octet-stream" can be handled.
My logic is to post progress to user, so that UI can be updated indicating user progress in upload, below my logic for same.
int toRead = length; // File length
byte[] data = new byte[4096];
connection.setDoOutput(true);
if(length != -1) {
connection.setFixedLengthStreamingMode(length);
} else {
connection.setChunkedStreamingMode(4096);
}
OutputStream os;
int i;
int count;
os = connection.getOutputStream();
int progress= 0;
try {
for(i = 0; (count= is.read(data)) > 0; ++i) { // is, is not null and contains a valid input stream
os.write(data, 0, count); // at this line am getting unexpected end of stream
progress+= count;
if(i % 20 == 0) {
rs.deliverProgress(progress, 0L);
progress= 0;
}
}
os.flush();
} finally {
if(is != null) {
is.close();
}
if(os != null) {
os.close();
}
}
on executing above code am getting this, although I have verified, output stream is not null, neither do input stream, it fails in first iteration of read loop itself, am seeing it has read 4096 bytes and then trying to write the same.
java.net.ProtocolException: unexpected end of stream
at com.android.okhttp.internal.http.HttpConnection$FixedLengthSink.close(HttpConnection.java:326)
at com.android.okio.RealBufferedSink.close(RealBufferedSink.java:174)
at com.android.okio.RealBufferedSink$1.close(RealBufferedSink.java:142)
any help in debugging above will he highly appreciated.
This may help you :
That exception is thrown by FixedLengthInputStream when the expected number of bytes (usually set in the content-length header of the response) is larger than the actual data in the response.
Check that the content-length header is correct. (If you're supplying your own value for the content length, make sure it is correct.)
It would help to see your code that sets up the input stream.
Already Fixed it, please add "Accept-Encoding", "identity" in header, then the server-side will get command that it will not modify the response, then send back to Clients.
If you have checked everywhere in your code and tried every solution in stackoverflow and github but the issue still occurs, and you have only tested your code on emulator, then, you should try to run your code on your real device instead. Maybe it will work, or maybe it won't, but if you feel desperate, just have a try, seriously. I was astonished when I happened to find that my code ran with bugs on emulator everytime but successfully on my mobile phone. Besides, the code also ran sucessfully on others' android emulators. So I guess there is something wrong in my android studio configuration that I can't find out. I have no idea why this happen, just like we don't know why "Clean Project/Invalidate caches" sometimes works better than any solution.
It is a little strange that your data length might be unknown.
Is it a media file? Or a live stream?
Anyway, I tried to upload my live stream data. And it happened in the same error.
I added this setting to the Connection and solved my problem.
Transfer-Encoding : chunked
("setChunkedStreamingMode" didn't work. I still don't know why.)
This happens for me on android emulator and doesn't happen on my physical android device.
I was doing GET request to flask server running on 0.0.0.0 on my laptop from the android app.
To fix it on the emulator, add the servers ip address in the emulators proxy.
see How to set up Android emulator proxy settings
The exact problem i had was unexpected end of stream retrofit
Parse for Android: Trying to get a device token in Parse but it keeps returning null. This code was working about 6 months back but lately have noticed this issue. Using the device token to subscribe to Parse later on. It just gets stuck in the while loop.I am using Parse 1.7.1 version. Even if I update the parse will this be the right way to get the device token?
private static final String KEY_DEVICE_TOKEN = "deviceToken";
boolean isTokenReady = false;
while (!isTokenReady) {
String deviceToken = (String) ParseInstallation.getCurrentInstallation().get(KEY_DEVICE_TOKEN);
if (!StringHelper.isNullOrEmpty(deviceToken)) {
isTokenReady = true;
} else {
sleep(1000);
}
}
ParsePush.subscribeInBackground("pushtoken_" + deviceToken);
You can use this, if you are retrieving a String:
ParseInstallation.getQuery().get(objectId).getString(KEY_DEVICE_TOKEN)
If you need to get the objectId from the default installation class:
ParseInstallation.getCurrentInstallation().getObjectId();
I'm using version 1.9.2. Hope this helps!
There's been 11 updates of the Android parse sdk. I would definitely update since there's lots of fixes.
Also, you shouldn't have to block your thread to wait for the device token. Did you forgot to save the installation before trying to get the deviceToken?
Like this:
ParseInstallation.getCurrentInstallation().save();
String deviceToken = (String) ParseInstallation.getCurrentInstallation().get( "deviceToken" );
Lastly, Im not sure why you would use a unique device token as push channels. You can use the deviceToken directly. So I would suggest not to subscribe to any channels and push notifications to selected devices using their deviceTokens.
I spent a lot of time on this problem too...
getInstallationId() seems to work. I use installationId to query installations and now it works OK
I'm building an Application that integrates with your Nest devices (both the thermostat and the Nest Protect, but this issue is about the thermostat).
What I'm trying to do is set my thermostat's ETA to be in x minutes (2 hours for example so 120 minutes).
This is my code that I'm executing:
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSSZ");
final String path = buildStructureFieldPath(structureID, Keys.STRUCTURE.ETA);
Structure.ETA eta = new Structure.ETA.Builder()
.setTripID(tripId)
.setEstimatedArrivalWindowBegin(sdf.format(estimatedArrivalBegin))
.setEstimatedArrivalWindowEnd(sdf.format(estimatedArrivalEnd))
.build();
sendRequest(path, eta.toJSON().toString(), listener);
The path is /structures/MY_STRUCTURE_ID/eta
Unfortunately that's not working. I'm always getting an error code -2 and error message: No write permission(s) for field(s): eta
And that's were it gets strange. No permission, but I did request the permission and I did an authenticate, which is successful, before launching the update call.
In the two attached screenshots you can see first my Nest Developer Account where you can find the ETA write permission and in the second you can see the logging from within my app (using the NestAPI as can be found on GitHub, just added the ETA feature myself).
Anyone have any idea on how to solve this issue?
Can you print out the exact JSON blob you're sending and post it here? (the value of eta.toJSON().toString())
Best guess is that it isn't formatted exactly correctly and as such is maybe attempting to write in such a way that doesn't adhere to the api-reference.
This is the format that it needs to match:
"eta": {
"trip_id": "myTripHome1024" ,
"estimated_arrival_window_begin": "2015-10-31T22:42:59.000Z" ,
"estimated_arrival_window_end": "2015-10-31T23:59:59.000Z"
}
Single line:
{"eta":{"trip_id":"myTripHome1024","estimated_arrival_window_begin":"2015-10-31T22:42:59.000Z","estimated_arrival_window_end": "2015-10-31T23:59:59.000Z"}}
To pinpoint exactly which field may be erroneous, try sending just one change at a time for each ie: structures/ID/eta/trip_id, etc for the others.
Useful JSON Validator: http://jsonlint.com/
You could also try to send it to /structures/MY_STRUCTURE_ID.json?auth=[TOKEN] instead of /structures/MY_STRUCTURE_ID/eta.
I'm using JDOM with my Android project, and every time I get a certain set of characters in my server response, I end up with these error messages:
05-04 10:08:46.277: E/PARSE: org.jdom.input.JDOMParseException: Error on line 95 of document UTF-8: At line 95, column 5263: unclosed token
05-04 10:08:46.277: E/Error Handler: Handler failed: org.jdom.input.JDOMParseException: Error on line 1: At line 1, column 0: syntax error
When I make the same query through google chrome, I can see that all of the XML came through fine, and that there are in fact no areas where a token is not closed. I have run into this problem several times throughout the development of the application, and the solution has always been to remove odd ascii characters (copyright logos, or trademark characters, etc. that got copied/pasted into those data fields). How can I get it to either a remove those characters, or b strip them and continue the function. Here's an example of one of my parse functions.
public static boolean parseUserData(BufferedReader br) {
SAXBuilder builder = new SAXBuilder();
Document document = null;
try {
document = builder.build(br);
/* XML Output to Logcat */
if (document != null) {
XMLOutputter outputter = new XMLOutputter(
Format.getPrettyFormat());
String xmlString = outputter.outputString(document);
Log.e("XML", xmlString);
}
Element rootNode = document.getRootElement();
if (!rootNode.getChildren().isEmpty()) {
// Do stuff
return true;
}
} catch (Exception e) {
GlobalsUtil.errorUtil
.setErrorMessage("Error Parsing XML: User Data");
Log.e(DEBUG_TAG, e.toString());
return false;
}
}
It distinctly sounds like a character encoding issue. I think duffymo is correct in his assessment. I have two comments though ....
If you are getting your data through a URL you should be using the URLConnection.getContentType() to get the charset (if it is set and the charset is not null) to set up the InputStreamReader on the URL's InputStream...
Have you tried JDOM 2.0.1? It is the first JDOM version that is fully tested on Android... (and the only 'supported' JDOM version on Android). JDOM 2.0.1 also has a number of performance tweaks, and memory optimizations that should make your processing faster. It also fixes a number of bugs.... though from what I see you should not run in to any bug problems.....
Check out https://github.com/hunterhacker/jdom/wiki/JDOM2-Migration-Issues and https://github.com/hunterhacker/jdom/wiki/JDOM2-and-Android
Is the BufferedReader constructed to take the encoding argument? Perhaps you need to tell the Reader or InputStream that you pass to use UTF-8.
I am trying out odata4j in my android app to retrieve data from a DB that can be accessed from a WCF service.
ODataConsumer co = ODataConsumer.create("http://xxx.xx.xx.xxx:xxxx/Users");
for(OEntity user : co.getEntities("Users").execute())
{
// do stuff
}
However this crashes at the call to getEntities. I have tried a variety of other calls as well, such as
Enumerable<OEntity> eo = co.getEntities("Users").execute();
OEntity users = eo.elementAt(0);
However this also crashes at eo.elementAt(0).
The logcat doesn't tell me anything, and the callstack seems to be Suspended at ActivityThread.performLaunchActivity.
Entering "http://localhost:xxxx/Users" in my web browser on the other hand works as expected and returns the users in my DB in xml format.
Any ideas on how I can debug this?
To log all http requests/responses:
ODataConsumer.dump.all(true);
The uri passed to the consumer .create call should be the service root. e.g. .create("http://xxx.xx.xx.xxx:xxxx/"); Otherwise your code looks fine.
Note the Enumerable behaves like the .net type - enumeration is deferred until access. If you plan on indexing multiple times into the results, I'd suggest you call .toList() first.
Let me know what you find out.
Hope that helps,
- john
I guess the call should be:
ODataConsumer co = ODataConsumer.create("http://xxx.xx.xx.xxx:xxxx");
for(OEntity user : co.getEntities("Users").execute())
{
// do stuff
}
create defines service you want to connect but Users is the resource you want to query.
Can you try this way.
OEntity oEntity;
OQueryRequest<OEntity> oQueryRequest= oDataJerseyConsumer.getEntities(entityName);
List<OEntity> list= oQueryRequest.execute().toList();
for (OEntity o : list) {
List<OProperty<?>> props = o.getProperties();
for (OProperty<?> prop : props) {
System.out.println(prop.getValue().toString());
}
}