Develop Reference

Realm: can't parse 110k objects on real android device

Essentially the question was in title. In detail: I am using Realm for storing
users in device storage. I am doing one request to server which returns list with 110k objects, gson parses this, all actions from out of memory were done. On emulator this operation continues for about 3 minutes and all saved to db successfully. All json data is valid, if it wasn't, on emulator it would crashed too. But, when I checked on real android device I got this
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Expected ':' at line 2 column 3 path $[70816].info2000
when gson was trying to parse 70816 object it couldn't, because it's not valid. I re-checked this user in database. All was good. I ran again and got this exception on another object
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Expected name at line 2 column 1091 path $[56000].accountCrm
So it's randomly.
I got confused and began researching, and I couldn't find anything. These objects come in response body(okhttp), I parse it like this:
Gson gson = GsonBase.getGlobalGson();
JsonReader reader = new JsonReader(responseBody.charStream()); //these are my 110k objects
reader.setLenient(true);
reader.beginArray();
List<ContactCRMRealm> contactCRMRBuffer = new ArrayList<>();
int counter = 0;
while (reader.hasNext()) {
if (counter < BUFFER_SIZE) {
contactCRMRBuffer.add(gson.fromJson(reader, ContactCRMRealm.class));
counter++;
} else {
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
counter = 0;
contactCRMRBuffer.clear();
}
}
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
contactCRMRBuffer.clear();
reader.endArray();
Exceptions are thrown in this line:
contactCRMRBuffer.add(gson.fromJson(reader, ContactCRMRealm.class));
I can't figure out but It seems like response body is being broken unexpectedly. Why It works on emulator and doesn't on android device? Few things that I can say: when I comment this line in else block:
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
all parsing side was performed without exceptions. I think it's because of realm. But what exactly? I re-wrote this parsing using createOrUpdateAllFromJson() realm method:
defaultInstance.beginTransaction();
try {
defaultInstance.createOrUpdateAllFromJson(ContactCRMRealm.class, responseBody.byteStream());
defaultInstance.commitTransaction();
} catch (IOException e) {
defaultInstance.close();
} finally {
defaultInstance.close();
}
On emulator was out of memory(wtf), on real device nothing happened for 10 minutes and login exception was thrown(read timeout 10 minutes).Also I was trying to make offset, by 10 thousands users on one request, nothing changed. Please help.

Ok, after a day of investigation, the problem was solved. Realm was 2.3.0 version in the app(yes, really old), so I was thinking maybe need to upgrade to newer verison. I did it, I upgraded realm to 3.7.0(not the last one because rxjava1 is using in the app). That's it. There are a lot of improvements, especially speed improvements, so now all is fine.

AWS DynamoDB Scan failing on iOS but works on Android

I am using DynamoDB as back-end database for my mobile app, and the schema etc are identical across Android & iOS. For a particular use-case, I have to perform a Scan, based on two attributes which are not indexed. For iOS Objective C, I am using the following code:
AWSDynamoDBScanExpression *scanExpression = [AWSDynamoDBScanExpression new];
scanExpression.limit = [NSNumber numberWithInt:maxCount];
scanExpression.filterExpression = #"#l = :location AND event = :event";
scanExpression.expressionAttributeNames = #{#"#l":#"location"};
scanExpression.expressionAttributeValues = #{#":location":location,
#":event":EVENT_TASTING};
Both location and event are Strings. EVENT_TASTING is a String constant. This scan keeps returning zero results, even though I have validated that for the provided entries I should be receiving the results. I use the following code in Android Java:
DynamoDBScanExpression scanExpression = new DynamoDBScanExpression();
scanExpression.setLimit(maxCount);
scanExpression.addFilterCondition("location",
new Condition()
.withComparisonOperator(ComparisonOperator.EQ)
.withAttributeValueList(new AttributeValue().withS(location)));
scanExpression.addFilterCondition("event",
new Condition()
.withComparisonOperator(ComparisonOperator.EQ)
.withAttributeValueList(new AttributeValue().withS(Constants.EVENT_TASTING)));
The scan works as expected in Android. What needs to change in iOS to make it work there too? I updated iOS SDK to 2.3.6 but it has not made a difference. This is the only scan operation I am doing in my code.
Is there an error in my scanExpression for iOS? Is there a way I can use the Android-style syntax to make this work on iOS?
Update
I tried the following changes:
AWSDynamoDBScanExpression *scanExpression = [AWSDynamoDBScanExpression new];
AWSDynamoDBAttributeValue *locationVal = [AWSDynamoDBAttributeValue new];
locationVal.S = location;
AWSDynamoDBAttributeValue *eventVal = [AWSDynamoDBAttributeValue new];
eventVal.S = EVENT_TASTING;
scanExpression.limit = [NSNumber numberWithInt:maxCount];
scanExpression.filterExpression = #"#l = :location AND event = :event";
scanExpression.expressionAttributeNames = #{#"#l":#"location"};
scanExpression.expressionAttributeValues = #{#":location":locationVal,
#":event":eventVal};
But now I am getting an error:
The request failed. Error: [Error Domain=com.amazonaws.AWSDynamoDBErrorDomain Code=0 "(null)" UserInfo={message=ExpressionAttributeValues contains invalid value: Supplied AttributeValue is empty, must contain exactly one of the supported datatypes for key :location, __type=com.amazon.coral.validate#ValidationException}]

Thanks to the hint from #YosukeMatsuda, I was able to fix this by calling Scan repeatedly until LastEvaluatedKey is empty. I am posting this as answer because unfortunately Mike's answer is not pointing out the correct issue and is misleading.
Here's how I changed the code in iOS:
// In a different method (for first call):
AWSDynamoDBScanExpression *scanExpression = // See code in original question
// In a new method that can be called recursively:
// DynamoDBObjectMapper scan:class-for-model expression:scanExpression
// continueWithBlock -> if (task.result):
AWSDynamoDBPaginatedOutput *paginatedOutput = task.result;
if (paginatedOutput.items.count != 0)
// Append the paginatedOutput.items to the cumulative array
else
// Replace the cumulative array with paginatedOutput.items
if (paginatedOutput.lastEvaluatedKey.count == 0) {
// Scan is complete - handle results
} else {
// Check if you have sufficient results
// In my case I had asked for 25 results but was getting 39
// So it doesn't seem to obey the scanExpression.limit value
// If more results are needed, continue the scan
[scanExpression setExclusiveStartKey:paginatedOutput.lastEvaluatedKey];
// Call this method recursively
}
If there is a more elegant solution I'd love to hear it. But at least it works now.

There are several differences between the Android code you're using and the ObjectiveC version.
in the Android Version you're using the older Filter Condition API while in the ObjectiveC you're using the more modern Filter Expression API; this doesn't necessarily make the newer one fail but it's just something to point out
in the case of ExpressionAttributeValues, the values for location and event that you're passing in should be of type AWSDynamoDBAttributeValue *, not strings; if you make this change your query will most likely start working.
I hope this answers your question but can't be certain because you only say "this works as expected in Android - how can I make it work in iOS" but you're not telling us what's broken.

Android support of dictionary based BreakIterator

I have the following same code run in API level 16 vs API level 21, and I found that in API level 16, the dictionary based iterator (tokenizer) seems not working, while in API level 21, the dictionary based iterator is working properly.
BreakIterator it = BreakIterator.getWordInstance();
String txt = "我们一起";
it.setText(txt);
int start = it.first();
int end = it.next();
buf = new StringBuffer();
while (end != BreakIterator.DONE) {
String word = txt.substring(start,end).trim();
if (!word.isEmpty()) {
buf.append(word);
buf.append("+");
}
start = end;
end = it.next();
}
vw.setText(buf);
In API Level 21, the text view shows ("我们" is a word, "一起" is a word)
我们+一起+
However in API Level 16, it shows as below (each Chinese character is a word):
我+们+一+起+
So I suspect that the API level 21 has enabled the dictionary based iterator, while previous API versions not.
However, after I have a search in the C++ source code of Android, I found that the key function RuleBasedBreakIterator::checkDictionary is both there in rbbi.cpp, for both API levels. It gives me the hints that both API shall support dictionary based iterator. I also suspect that the difference is because of the different category value set for different char-set. However I am not able trace back how these values are set and whether there is difference.
My question is, how to further confirm that the API implementation is enhanced in API level 21?

How to ignore diacritics with a regex in Android 2.2?

I need to use Android 2.2 (API 8) and I have seen that Normalizer.normalize(text, Normalizer.Form.NFD) is only available from Android API 9.
What can I add to my regex to ignore accents ?
SpannableStringBuilder stringBuilder = new SpannableStringBuilder(text);
for (String keyword : keywords.keySet()) {
String caseInsensitiveRegex = "(?i)\\b" + keyword + "\\b";
Pattern pattern = Pattern.compile(caseInsensitiveRegex);
Matcher m = pattern.matcher(stringBuilder);
while (m.find()){
// Do some stuff
}
}
Thanks per advance for your help ;)

is only available from Android API 9
Android is open sourced project. There's nothing preventing you from grabbing that part of API 9 and putting directly in your app so you will no longer be API dependent and I at the moment I am not seeing any obstacles for doing this in your case.

JSONObject.toString() gives different result in 2.2 than in 2.1 (.Net Date format)

In Android 2.1 this
JSONObject o = new JSONObject();
o.put("MyDate", "/Date(1289334937639)/");
Log.d(TAG, o.toString());
produces
{"MyDate":"/Date(1289334937639)/"}
but in 2.2 it produces
{"MyDate":"\/Date(1289334937639)\/"}
I am talking to a .Net web service so the 2.2 version works correctly for me. How do I make 2.1 produce the same thing without breaking 2.2?
Thanks for your help.

Get the latest version of of JSON from http://www.json.org/java/ and integrate it to your code. You just need to change your imports I guess.

I ended up with the following:
if (Build.VERSION.SDK_INT == 7) {
params = params.replaceAll("/", "\\\\/");
}
where params is the json already converted to a string. Ugly, but it works.