Develop Reference

Realm: can't parse 110k objects on real android device

Essentially the question was in title. In detail: I am using Realm for storing
users in device storage. I am doing one request to server which returns list with 110k objects, gson parses this, all actions from out of memory were done. On emulator this operation continues for about 3 minutes and all saved to db successfully. All json data is valid, if it wasn't, on emulator it would crashed too. But, when I checked on real android device I got this
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Expected ':' at line 2 column 3 path $[70816].info2000
when gson was trying to parse 70816 object it couldn't, because it's not valid. I re-checked this user in database. All was good. I ran again and got this exception on another object
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Expected name at line 2 column 1091 path $[56000].accountCrm
So it's randomly.
I got confused and began researching, and I couldn't find anything. These objects come in response body(okhttp), I parse it like this:
Gson gson = GsonBase.getGlobalGson();
JsonReader reader = new JsonReader(responseBody.charStream()); //these are my 110k objects
reader.setLenient(true);
reader.beginArray();
List<ContactCRMRealm> contactCRMRBuffer = new ArrayList<>();
int counter = 0;
while (reader.hasNext()) {
if (counter < BUFFER_SIZE) {
contactCRMRBuffer.add(gson.fromJson(reader, ContactCRMRealm.class));
counter++;
} else {
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
counter = 0;
contactCRMRBuffer.clear();
}
}
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
contactCRMRBuffer.clear();
reader.endArray();
Exceptions are thrown in this line:
contactCRMRBuffer.add(gson.fromJson(reader, ContactCRMRealm.class));
I can't figure out but It seems like response body is being broken unexpectedly. Why It works on emulator and doesn't on android device? Few things that I can say: when I comment this line in else block:
defaultInstance.executeTransaction(realm1 -> realm1.insertOrUpdate(contactCRMRBuffer));
all parsing side was performed without exceptions. I think it's because of realm. But what exactly? I re-wrote this parsing using createOrUpdateAllFromJson() realm method:
defaultInstance.beginTransaction();
try {
defaultInstance.createOrUpdateAllFromJson(ContactCRMRealm.class, responseBody.byteStream());
defaultInstance.commitTransaction();
} catch (IOException e) {
defaultInstance.close();
} finally {
defaultInstance.close();
}
On emulator was out of memory(wtf), on real device nothing happened for 10 minutes and login exception was thrown(read timeout 10 minutes).Also I was trying to make offset, by 10 thousands users on one request, nothing changed. Please help.

Ok, after a day of investigation, the problem was solved. Realm was 2.3.0 version in the app(yes, really old), so I was thinking maybe need to upgrade to newer verison. I did it, I upgraded realm to 3.7.0(not the last one because rxjava1 is using in the app). That's it. There are a lot of improvements, especially speed improvements, so now all is fine.

Receiving hexadecimal values from socket instead of strings on SocketServer

I opened a socket between an Android app and a python server. The combination is that the Server listens, and android connects to the Server.
Here is the server code. The problematic part takes place in the definition of handle :
import SocketServer
from time import sleep
import sys
HOST = '192.168.56.1'
PORT = 2000
class SingleTCPHandler(SocketServer.StreamRequestHandler):
def handle(self):
try:
while(1):
sleep(0.03)
data = self.rfile.readline().strip()
print data
except KeyboardInterrupt:
sys.exit(0)
class SimpleServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
allow_reuse_address = True
def __init__(self, server_address, RequestHandlerClass):
SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass)
server = SimpleServer((HOST, PORT), SingleTCPHandler)
try:
server.serve_forever()
except KeyboardInterrupt:
sys.exit(0)
The connection is established normally, and the Android app sends the following data to the socket:
'0:0'
But the data is received on the Server as:
'\x000\x00:\x000\x00'
The variable that receives the data is:
data = self.rfile.readline().strip()
and printing gives the regular format:
In [2]: print data
0:0
I didn't manage to step into the print function with pdb to see what it does.
I'm looking for a way to convert the '\x000\x00:\x000\x00' to '0:0'.
Please advise on a way to convert the variable. You are welcome to comment/criticize the whole implementation. This is my first project in dealing with sockets so i don't know the pitfalls.
Update
This was the original Android code:
String podaci = "0:0";
public void Socketic() throws IOException {
Socket mojSocket = new Socket(urlServer, port);
DataOutputStream izlazdata = new DataOutputStream(
mojSocket.getOutputStream());
while (podaci != "end") {
try {
Thread.sleep(60);
} catch (InterruptedException e) {
e.printStackTrace();
}
izlazdata.writeChars(podaci);
izlazdata.flush();
}
izlazdata.close();
mojSocket.close();
};
And the problem was, as you suspected in:
izlazdata.writeChars(podaci);
writeChars uses the method writeChar. The API documentation for writeChar states:
Writes a char to the underlying output stream as a 2-byte value, high byte first...
The two bytes represent the 16bits which UTF-16 uses for encoding.
When we changed it to everything started working:
izlazdata.writeBytes(podaci);
Update
Based on the answers given, here is how the unwanted string is to be interpreted in terms of characters.
This solves my concrete problem, however, if someone would give a more generic solution to what happend here so that a larger lesson can be learned.
If not, i will accept Esailijas answer in a few days.

You need to show the code happening Android but it strongly seems like it's sending data in UTF-16BE. You should specify the encoding on the Android end. The characters are not hexadecimal literally, but because the NUL character is unprintable, python shows \x00 instead.
Another option is to decode it:
self.rfile.readline().decode("utf_16_be").strip()
note that the result of this is an unicode string.

Difficulty Reading XML File

Using Eclipse 3.7.2
Android 2.3.3 / API Level 10
I am having trouble reading an XML file. I "think" I get a proper handle on the file with this code and open it for input / output as I will want to read the info, display for user, then update with any changes.
Coming from the c# world when I goto read the file I see a behavior I am unfamiliar with. In the debug perspective my variables have red square values. Now this may be OK but my first thought is red == trouble? However the data (the data in the variables window of the debug perspective) seems ok so I proceed. My eventType is a zero and I enter into the case statement but then I throw an exception. Now here I drop the ball. I don't know how to get more information about exceptions in Java. For example in the c# world I could get the inner exception. I'm sure there is a way in Java I just haven't learned yet.
Here are the steps:
From line 1 I jump straight to line 9. F6 (Eclipse debug shortcut for "Step over") takes me to line 4, F6 again back to line 9, F6 a final time and I goto my catch for a generic exception.
try
{
DataFileOut = new FileOutputStream(LiftFile);
DataFileIn = new FileInputStream(LiftFile);
InputStreamReader isr = new InputStreamReader(DataFileIn);
// auto-detect the encoding from the stream
parser.setInput(isr);
int eventType = parser.getEventType();
Category currentCategory = null;
boolean done = false;
while (eventType != XmlPullParser.END_DOCUMENT && !done)
{
String name = null;
0 //eventType = 2;
1 switch (eventType)
{
3 case XmlPullParser.START_DOCUMENT:
4 break;
case XmlPullParser.START_TAG:
....
}
9 eventType = parser.next();
}
}
So my thoughts are I have a bad FileInputStream handle or my XML document is not well formed?? Any other thoughts?
<?xml version="1.0" encoding="UTF-8"?>
<ExcersiseInformation>
<Lifts count="6">
<Lift>
<ID>1001001</ID>
<Name>Hammer Curls</Name>
<PreviousLift>45</PreviousLift>
<PreviousReps>8</PreviousReps>
<SuggestLift>45</SuggestLift>
<SuggestReps>10</SuggestReps>
<ActualLift></ActualLift>
<ActualReps></ActualReps>
</Lift>
.....
</Lifts>
</ExcersiseInformation>

Your xml seems to be well formed. Are you using a handler and parser class to get xml tag contents? Probably best method to reach it. You can find some good examples on internet or other posts. Please take a look to following one, same method that I am using successfuly
http://www.jondev.net/articles/Android_XML_SAX_Parser_Example

how to ignore XML errors when parsing faulty XML with Sax (on Android)

Question on the Sax XML parser on Android, using Java: I need to parse XML files I get from the web, and that I have no control over. Some contain errors and cause the parser to abort with errors like "mismatched tag" or "not well-formed (invalid token)".
Those errors don't matter to me, I want to ignore them and keep going, I can handle the broken XML structure. But I cannot fix the XML files, they are not mine. How can I tell Sax on Android (class org.xml.sax.XMLReader) to not throw an exception and keep going? Attaching an ErrorHandler didn't work, and catching the exception is of no use because I can't resume parsing where it stopped.
My XML is not HTML, but here are some (X)HTML examples where browsers ignore errors and keep going. I want to do this too.
Browsers are fine with "<br>" instead of "<br/>" even though the tag is never closed.
"<b><i> text </b></i>" works even though the closing tags are in the wrong order.
"odds & ends" is accepted despite the invalid token, "odds & ends" would be correct.
I'd prefer to not write my own parser, dealing with character set conversions and all that. I don't need to validate XML. Here's my code, reduced to the essentials:
XMLReader r = SAXParserFactory.newInstance().newSAXParser().getXMLReader();
r.setErrorHandler(new MyLenientErrorHandlerThatNeverThrows());
r.setContentHandler(new MyImporterThatExtendsDefaultHandler());
r.parse(new InputSource(new BufferedReader(...)));
Thanks!

Ok, it appears it can't be done. Sax supports error detection but not error recovery, which makes it less than ideal for robust code in this example.
Got it to work by replaxing Sax with XmlPullParser, which allows wrapping the next-token call in a try-catch block:
try {
XmlPullParserFactory factory = XmlPullParserFactory.newInstance();
XmlPullParser xpp = factory.newPullParser();
xpp.setInput(in);
int type = xpp.getEventType();
while (type != XmlPullParser.END_DOCUMENT) {
switch (type) {
case XmlPullParser.START_TAG: startTag(xpp); break;
case XmlPullParser.END_TAG: endTag(xpp); break;
case XmlPullParser.TEXT: characters(xpp.getText()); break;
}
try {type = xpp.next();}
catch (XmlPullParserException e) {}
}
} catch (Exception e) {}

Stream android logcat output to an sd card

I want to achieve the following but so far, no luck
Open a file in the SD Card when the android application first started.
Stream the logcat output to the file.
When the application exits, stop the logcat streaming.
In my ApplicationClass extends Application onCreate() method, I do this
wwLogManager.openLogFile();
and here's the code in the openLogFile()
private static final String LOGCAT_COMMAND = "logcat -v time -f ";
String timestamp = Long.toString(System.currentTimeMillis());
String logFileName = BuildConstants.LOG_LOCATION + "_" + timestamp + ".log";
mLogFile = new File(Environment.getExternalStorageDirectory() + logFileName);
mLogFile.createNewFile();
String cmd = LOGCAT_COMMAND + mLogFile.getAbsolutePath();
Runtime.getRuntime().exec(cmd);
I do get log files in the sd card, but the log output in these files do not have any trace of the Log.i() calls that I placed in my activities. Is the logcat command that I used here correct? Thanks!

I apologize if I am misunderstanding your goals, but perhaps you could use the java.util.logging API instead of using Logcat or the Android Logging mechanism.
Like the Android logging API, the java.util.logging API allows you to easily log messages at various levels, such as FINE, FINER, WARN, SEVERE, etc.
But the standard logging API has additional advantages, too. For example, you can easily create a log file by using a FileHandler. In fact, FileHandler has a built-in log rotation mechanism, so you don't have to worry (so much) about cleaning up the log files. You can also create a hierarchy of Loggers; so, for example, if you have two Loggers, com.example.foo and com.example.foo.bar, changing the logging level of the former will also change the logging level of the latter. This will even work if the two Loggers are created in different classes! Moreover, you change logging behavior at runtime by specifying a logging configuration file. Finally, you can customize the format of the log by implementing your own Formatter (or just use the SimpleFormatter to avoid the default XML format).
To use the standard logging API, you might try something like this:
// Logger logger is an instance variable
// FileHandler logHandler is an instance variable
try {
String logDirectory =
Environment.getExternalStorageDirectory() + "/log_directory";
// the %g is the number of the current log in the rotation
String logFileName = logDirectory + "/logfile_base_name_%g.log";
// ...
// make sure that the log directory exists, or the next command will fail
//
// create a log file at the specified location that is capped 100kB. Keep up to 5 logs.
logHandler = new FileHandler(logFileName, 100 * 1024, 5);
// use a text-based format instead of the default XML-based format
logHandler.setFormatter(new SimpleFormatter());
// get the actual Logger
logger = Logger.getLogger("com.example.foo");
// Log to the file by associating the FileHandler with the log
logger.addHandler(logHandler);
}
catch (IOException ioe) {
// do something wise
}
// examples of using the logger
logger.finest("This message is only logged at the finest level (lowest/most-verbose level)");
logger.config("This is an config-level message (middle level)");
logger.severe("This is a severe-level message (highest/least-verbose level)");
The Android logging mechanism is certainly easy and convenient. It isn't very customizable, though, and log filtering must be done with tags, which can easily become unwieldy. By using the java.uitl.logging API, you can avoid dealing with a multitude of tags, yet easily limit the log file to specific parts of your application, gain greater control over the location and appearance of the log, and even customize logging behavior at runtime.

I repost my answer here so #JPM and others can see... The code basically just execute the logcat command and then build the log from the input stream.
final StringBuilder log = new StringBuilder();
try {
ArrayList<String> commandLine = new ArrayList<String>();
commandLine.add("logcat");
commandLine.add("-d");
ArrayList<String> arguments = ((params != null) && (params.length > 0)) ? params[0] : null;
if (null != arguments){
commandLine.addAll(arguments);
}
Process process = Runtime.getRuntime().exec(commandLine.toArray(new String[0]));
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = bufferedReader.readLine()) != null){
log.append(line);
log.append(LINE_SEPARATOR);
}
}
catch (IOException e){
//
}
return log;

Try manually setting a filter as described here:
http://developer.android.com/guide/developing/debugging/debugging-log.html#filteringOutput
Something like:
logcat ActivityManager:I MyApp:V *:S
If you replace "MyApp" with the log tags that you are using, that should show you all info (and greater) logs from ActivityManager, and all verbose (and greater) logs from your app.

I know this is a late answer to the question but I would highly recommend using Logback to write messages to a log file as well as the logcat.
Copy logback-android-1.0.10-2.jar and slf4j-api-1.7.5.jar to your libs folder.
Right-Click on both libs and from the menu select Build Path -> Add to Build Path.
Create a logback.xml file in your assets folder and enter the following:
%msg
WARN
${LOG_DIR}/log.txt
%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
To write a log:
public class MyActivity extends Activity
{
public static final Logger logger = LoggerFactory.getLogger(MyActivity.class);
protected void onCreate(Bundle b)
{
super(b);
try{
throw new Exception("Test");
}catch(Exception e){
logger.error("Something bad happened",e);
}
}
}