I want to send the log lines to email in every 10 minutes.
To do that, I have used a Timer and inside of timer I send the logs via email.
However I loose some log lines between 2 emails.
For example my first email contains no lines which is normal according to my algorithm.
My second email contains log lines between 15.37 and 15.38 seconds.
My third email contains logs in between 15.44 and 15.48 time intervals.
My fourth email contains logs in between 15.55 and 15.58 time intervals.
As you can see I loose some of my logs but I could not find a way to avoid that.
Following is my code in my service class:
#Override
public void onCreate() {
super.onCreate();
mTimer = new Timer();
mTimer.scheduleAtFixedRate(new TimerTask() {
#Override
public void run() {
sendLogFile();
}
}, 0, 1000 * 60 * 10 );
}
Inside of sendSupport method the second parameter is sent as a content of the log lines where logs is a static string variable.
private void sendLogFile() {
mInteractor.sendSupport("LOG FILE", "MSG"+logs, "SUBJECT"+ System.currentTimeMillis(), "",
result -> {
Timber.log(Log.DEBUG, "sendSupport Thread.currentThread().getName() " + Thread.currentThread().getName());
if (result.isSuccess) {
Timber.d("is sent");
writeLogFile();
} else {
Timber.d("is NOT sent");
}
}
);
}
private void writeLogFile()
{
try {
StringBuilder logBuilder = new StringBuilder();
process = Runtime.getRuntime().exec( "logcat -d");
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(process.getInputStream()));
String line;
while ((line = bufferedReader.readLine()) != null) {
logBuilder.append(line + "\n");
}
logs = logBuilder.toString();
} catch (IOException e) {
e.printStackTrace();
}
}
As a result I could not figure out how am I going to be able to get all logs in periodically in my email.
Thanks.
While the #Knossos answer is pointing the reason why some logs are missing, it doesn't suggest how to use that knowledge to reliably get the logs from users' phones, when you don't have an access to their devices to run some adb commands.
Here is what I suggest to do instead:
As you do already use Timber, just add some more power to it. Namely, add an additional LoggingTree that will save the logs into a file, instead of just posting them to Logcat. You can have many Timber Trees working simultaneously, so you can have both Logcat and File logs if needed.
Use the same timer to send an email message when and where needed. But, instead of using access to logcat -d, simply use the file where Timber have already written the logs. Don't forget to flush the stream before sending the email. In order not to send the same logs again and again, configure the FileTree in a way that it creates a new file every time when a previous one is sent (seta a new file name via a method call, for example).
Profit :)
To log into two different systems (Trees), you need to simply add one more to Timber:
Timber.plant(new Timber.DebugTree());
Timber.plant(new FileLoggingTree());
And here is an example of FileLoggingTree(source):
public class FileLoggingTree extends Timber.DebugTree {
private static Logger mLogger = LoggerFactory.getLogger(FileLoggingTree.class);
private static final String LOG_PREFIX = "my-log";
public FileLoggingTree(Context context) {
final String logDirectory = context.getFilesDir() + "/logs";
configureLogger(logDirectory);
}
private void configureLogger(String logDirectory) {
// reset the default context (which may already have been initialized)
// since we want to reconfigure it
LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
loggerContext.reset();
RollingFileAppender<ILoggingEvent> rollingFileAppender = new RollingFileAppender<>();
rollingFileAppender.setContext(loggerContext);
rollingFileAppender.setAppend(true);
rollingFileAppender.setFile(logDirectory + "/" + LOG_PREFIX + "-latest.html");
SizeAndTimeBasedFNATP<ILoggingEvent> fileNamingPolicy = new SizeAndTimeBasedFNATP<>();
fileNamingPolicy.setContext(loggerContext);
fileNamingPolicy.setMaxFileSize("1MB");
TimeBasedRollingPolicy<ILoggingEvent> rollingPolicy = new TimeBasedRollingPolicy<>();
rollingPolicy.setContext(loggerContext);
rollingPolicy.setFileNamePattern(logDirectory + "/" + LOG_PREFIX + ".%d{yyyy-MM-dd}.%i.html");
rollingPolicy.setMaxHistory(5);
rollingPolicy.setTimeBasedFileNamingAndTriggeringPolicy(fileNamingPolicy);
rollingPolicy.setParent(rollingFileAppender); // parent and context required!
rollingPolicy.start();
HTMLLayout htmlLayout = new HTMLLayout();
htmlLayout.setContext(loggerContext);
htmlLayout.setPattern("%d{HH:mm:ss.SSS}%level%thread%msg");
htmlLayout.start();
LayoutWrappingEncoder<ILoggingEvent> encoder = new LayoutWrappingEncoder<>();
encoder.setContext(loggerContext);
encoder.setLayout(htmlLayout);
encoder.start();
// Alternative text encoder - very clean pattern, takes up less space
// PatternLayoutEncoder encoder = new PatternLayoutEncoder();
// encoder.setContext(loggerContext);
// encoder.setCharset(Charset.forName("UTF-8"));
// encoder.setPattern("%date %level [%thread] %msg%n");
// encoder.start();
rollingFileAppender.setRollingPolicy(rollingPolicy);
rollingFileAppender.setEncoder(encoder);
rollingFileAppender.start();
// add the newly created appenders to the root logger;
// qualify Logger to disambiguate from org.slf4j.Logger
ch.qos.logback.classic.Logger root = (ch.qos.logback.classic.Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME);
root.setLevel(Level.DEBUG);
root.addAppender(rollingFileAppender);
// print any status messages (warnings, etc) encountered in logback config
StatusPrinter.print(loggerContext);
}
#Override
protected void log(int priority, String tag, String message, Throwable t) {
if (priority == Log.VERBOSE) {
return;
}
String logMessage = tag + ": " + message;
switch (priority) {
case Log.DEBUG:
mLogger.debug(logMessage);
break;
case Log.INFO:
mLogger.info(logMessage);
break;
case Log.WARN:
mLogger.warn(logMessage);
break;
case Log.ERROR:
mLogger.error(logMessage);
break;
}
}
}
The problem is that logcat -d is only delivering you the latest X bytes of data from the stream. You aren't guaranteed to get everything between 10 minute intervals.
In the best case, you get what you want. In the worst cases, you miss log data or log sections overlap (you get some from the previous dump too).
You can see this here: adb logcat -d | dd
...
03-27 11:36:27.474 791 22420 E ResolverController: No valid NAT64 prefix (147, <unspecified>/0)
03-27 11:36:27.612 3466 3521 I PlayCommon: [657] alsu.c(187): Successfully uploaded logs.
453+111 records in
499+1 records out
255863 bytes (256 kB, 250 KiB) copied, 0,136016 s, 1,9 MB/s
As you can see, it is clearly a 256 kB chunk that is pulled through logcat -d.
On the plus side, you can change that! If you look at adb logcat --help you can see options.
For example, if you use adb logcat -d -t '100000' | dd (all logs in the last 100000 seconds. I now have the following:
...
03-27 11:45:41.687 791 1106 I netd : bandwidthSetGlobalAlert(2097152) <0.90ms>
03-27 11:45:42.098 21897 23376 V FA : Inactivity, disconnecting from the service
2237+1558 records in
2879+1 records out
1474408 bytes (1,5 MB, 1,4 MiB) copied, 1,20785 s, 1,2 MB/s
1.5 MB of logs. You should be able to get all logs with this.
You log the timestamp of each logcat pull, then use that each time to determine the seconds since the last pull.
I hope that helps!
You cannot send an email from the device without user interaction or implementing the email function yourself that allows to do this without user interaction.
Most apps have some api endpoint to send the logs to.
Related
The issue is that System.currentTimeMillis() returns the wrong milliseconds with different ranges of time mainly within the future sometimes up to 6 months, but it varies from a few seconds to months.
The device that this is occurring on is a Tablet model Huawei M2-A201W on android 5.1.1 the kernel version is: **3.10.74-gdbd9055**
My first assumption was that the NTP was somehow messing up with the time but I have thousands of those tablets and some of them have no network connection, no SIM card so no GSM/3G/4G.
Im using the System.currentTimeMillis() to save in a column for a table for when was a row created in the local sqlite database.
This anomally happens very often(30% of each System.currentTimeMillis() call) on the tablets that I use.
As a workaround for using System.currentTimeMillis(), maybe you can let sqlite handle the timestamp creation and see if that solves your problem?
Define/Alter your "created"-column with timestamp default current_timestamp or with default(strftime('%Y-%m-%d %H:%M:%f', 'now')) if you need milliseconds as well, like this:
sqlite> create table my_table(id integer primary key autoincrement not null, name text, created timestamp default(strftime('%Y-%m-%d %H:%M:%f', 'now')) not null);
sqlite> insert into my_table(name) values ('MyTestRow1');
sqlite> insert into my_table(name) values ('MyTestRow2');
sqlite> select * from my_table;
1|MyTestRow1|2017-08-07 10:08:50.898
2|MyTestRow2|2017-08-07 10:08:54.701
When you don't have SIM card so no GSM/3G/4G, your phone can't update correct time based on the network provided time/zone.
So, devices with the network show the correct time, while other devices without the network can show incorrect time -- you have to manually set the correct time. System.currentTimeMilis() reads the time from your system. g But on power turn, the clock works.
Check if NTP (UDP port 123) is blocked by apps using Socket or DatagramSocket. Note: NTP applies to the scenario where clocks of all hosts or routers on the network must be the same. If your device switch to two (or more) different network and gets time updated from different sources, it can fluctuate time.
Ultimately, your system time is being changed, that's why it is fluctuating. If you manually System.currentTimeMilis() after you manually disable automatic date and time, I believe, it doesn't fluctuate (no anomaly). If this is the case then your Huewai tablet doesn't have bugs.
The java API call System.currentTimeMillis() in Android platform use the POSIX api gettimeofday to get the time in milli seconds. See here.
static jlong System_currentTimeMillis(JNIEnv*, jclass) {
timeval now;
gettimeofday(&now, NULL);
jlong when = now.tv_sec * 1000LL + now.tv_usec / 1000;
return when;
}
It assume every call to gettimeofday will be successful. I guess your problem might happen here.
It's better to check the return value of every API call and determine what to do next if error happen.
So I suggest a more reliable method in JNI with your own implementation like below. Call these POSIX APIs in order, if gettimeofday failed, call clock_gettime, if it failed again, call time.
struct timeval now;
if (gettimeofday(&now, NULL) != 0) {
struct timespec ts;
if (clock_gettime(CLOCK_REALTIME, &ts) == 0) {
now.tv_sec = ts.tv_sec;
now.tv_usec = ts.tv_nsec / 1000LL;
} else {
now.tv_sec = time(NULL);
now.tv_usec = 0;
}
}
jlong when = now.tv_sec * 1000LL + now.tv_usec / 1000LL;
__android_log_print(ANDROID_LOG_INFO, "TAG", "%lld", when);
What about fetching the timestamp natively by running "date +%s" command in Linux kernel?
Here, "+%s" is seconds since 1970-01-01 00:00:00 UTC. (GNU Coreutils 8.24 Date manual)
try {
// Run the command
Process process = Runtime.getRuntime().exec("date +%s");
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
// Grab the results
StringBuilder log = new StringBuilder();
String line;
while ((line = bufferedReader.readLine()) != null) {
log.append(line);
}
} catch (IOException e) {
e.printStackTrace();
}
If you print this,
Log.e("unix_time: ", "" + log.toString());
You'll get an Unix timestamp e.g. 1502187111
To convert it back to a date object, multiply by 1000, since java is expecting milliseconds,
Date time = new Date(Long.parseLong(log.toString()) * 1000);
Log.e("date_time: ", "" + time.toString());
This will give you a plain date format. e.g. Tue Aug 08 16:15:58 GMT+06:00 2017
Since you mentioned that most of the calls are getting the proper time and it only happens 30% of the cases, I would create an receiver for ACTION_TIME_CHANGED and ACTION_TIMEZONE_CHANGED intent broadcasts to find out when the time changes. Maybe this will give you a clue on what is changing the time.
Together with the ConnectivityManager you can detect if the device is connected and what type of a connection you have, maybe some connection is triggering the time change.
// init the register and register the intents, in onStart, using:
receiver = new BroadcastReceiver() {
#Override
public void onReceive(Context context, Intent intent) {
getNetworkInfo();
if (Intent.ACTION_TIME_CHANGED.equals(intent.getAction()))
Log.d(this.getClass().getName(), "Detected a time change. isWifiConn: " +
isWifiConn + " isMobileConn: " + isMobileConn);
if (Intent.ACTION_TIMEZONE_CHANGED.equals(intent.getAction()))
Log.d(this.getClass().getName(), "Detected a timezone change. isWifiConn: " +
isWifiConn + " isMobileConn: " + isMobileConn);
}
};
IntentFilter filters = new IntentFilter();
filters.addAction(Intent.ACTION_TIME_CHANGED);
filters.addAction(Intent.ACTION_TIMEZONE_CHANGED);
registerReceiver(receiver, filters);
Log.d(DEBUG_TAG, "Receiver registered");
// do not forget to unregister the receiver, eg. onStop, using:
unregisterReceiver(receiver);
//...
private void getNetworkInfo() {
ConnectivityManager connMgr = (ConnectivityManager)
getSystemService(Context.CONNECTIVITY_SERVICE);
NetworkInfo networkInfo = connMgr.getNetworkInfo(ConnectivityManager.TYPE_WIFI);
isWifiConn = networkInfo.isConnected();
networkInfo = connMgr.getNetworkInfo(ConnectivityManager.TYPE_MOBILE);
isMobileConn = networkInfo.isConnected();
Log.d(DEBUG_TAG, "Wifi connected: " + isWifiConn);
Log.d(DEBUG_TAG, "Mobile connected: " + isMobileConn);
}
Does this device have a real hardware RTC? I couldn't find a definitive answer from googling and reading spec sheets. A run of:
$ dmesg -s 65535 | grep -i rtc
from a shell should give you an answer. You should see something like this (will vary with chipset and kernel version):
[ 3.816058] rtc_cmos 00:02: RTC can wake from S4
[ 3.816429] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
[ 3.816510] rtc_cmos 00:02: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
If there is no message returned by grep (and you grepped against the whole kernel message buffer), then there's your answer. You have no clock to keep time on these devices. You will always need NTP and a working network connection to the Internet to keep such a device without a clock chip in sync with world time.
RTC: https://en.wikipedia.org/wiki/Real-time_clock
I'm communication with a server through a tcp socket connection, i'm able to read lines that ends with \n fine, however when the line is not terminated (ends in \n) i'm not able to read it. I tried the following but it didn't work and caused my app to freeze at startup:
private Socket socket;
private BufferedReader input;
public boolean isConnected;
#Override
public void onCreate()
{
try
{
socket = new Socket ("server.ip.add.ress", 23456);
input = new BufferedReader (new InputStreamReader (socket.getInputStream());
handshake();
isConnected = true;
}
catch // Handle IOException and UnknownHostException
}
// custom runnable to read availabe input from the server
private class MyRunnable implements Runnable
{
private volativle String value;
public String getValue()
{
return value;
}
#Override
public void run()
{
int count;
char[] buffer = new char[10]; // expected message 'username: '
try
{
count = input.read (buffer, 0, 10);
if (count > 0) value = new String (buffer);
}
catch // IOException
}
}
// when connection is established with server expect 'username: ' from
// the server and send the user name back to it
public void handshake()
{
MyRunnable runnable = new MyRunnable();
try
{
Thread thread = new Thread (runnable);
thread.start();
thread.join();
String greeting = runnable.getValue();
if (greeting.equals ("username: ")) // Send username back
}
catch // InterruptedException
}
why is it hanging? and how can i read a non terminated line?
Edit:
To clarify: The server sends the greeting message username: immediately after the connection is established with a client, the client wait for the greeting and send back it's username when received (that's what handshake() does), if no handshake the client disconnects otherwise it start listening for incoming messages. Because i need to know if handshake is complete before starting the listener i had to use Thread.join().
The problem: Thanks for the comments and answers below, it turned out that BufferedReader.read() blocks the thread and waits until something is sent from the server and if nothing is being sent it causes the app to hang, Therefor there's no way to find out if the line has ended.
The solution: In my specific situation i just wanted to know if a specific message is sent "username: " so i used read (buffer, 0, 10) to read exactly 10 characters (the length of "username: "), and because it blocks if nothing is sent i used Thread.join (1000) which waits only one second and then if nothing received i disconnect the client.
Why is it hanging?
This is what it is suppose to be. It will block the thread if no data is available to read. This is also why you want to put it in a background thread.
Can it not just return if nothing is available?
What you are looking for is ready(), which will tell you whether there is available data or not.
Indicates whether this reader is ready to be read without blocking.
Returns
true if this reader will not block when read is called, false if unknown or blocking will occur.
But you should be very careful when using this function. Because networking is a lot about timing. The fact that you don't have any data to read at this second doesn't necessary mean that it won't be any data in the next second.
So a better design of the server should be more or less as the following:
If the username is found, return the username
If the username is not found, return an error message to let the client side know that the username is not found
There's no need for the thread. Your goal is to wait until you've read what you've been waiting for. Why not just let read() perform the wait for you?
What you're struggling with is the classic problem of TCP communication: "when do I know that I've got everything the server sent?"
In your case, you're expecting to read bytes until the collection of bytes ends with "username: ". So, change your algorithm to perform 1 byte reads (filling a buffer as you go) until that buffer ends with "username: ".
You can make a more complicated algorithm -- which would be more efficient -- that would attempt to read multiple bytes at a time and append them to a buffer -- performing your check each time. But either strategy is logically equivalent.
I also recommend just using the InputStreamReader. It has various read() methods. I am a bit suspicious about the BufferedInputReader, especially when dealing with data that isn't newline terminated. I'm probably just paranoid. I've just never used it when writing TCP client/server programs, so I'm not sure.
I'm trying to get data from a monitor to an Android application and I've took the IHE - PCD-01 transaction as a model.
The scheme is simple, is based on achieve the interconnection between the monitor and the tablet, where the monitor sends constantly information and the application is listening.
But what I don't understand is if I need an ACK or not after every message. Does anyone can help me with this?
TL;DR yes, nothing special here, support the usual HL7 ACK/NACK driven by MSH-15, MSH-16 fields. ACK-ing everything by default is "better safe then sorry"
The document "IHE Patient Care Device (PCD), Technical Framework, Volume 2 (PCD TF-2) Transactions, Revision 1.0 - Final Text, August 12, 2011" available at http://www.ihe.net/technical_framework/upload/ihe_pcd_tf_vol2_ft_2011-08-12.pdf says
..The common static definition of the HL7 acknowledgement (ACK) message is described in Appendix G, "HL7 Implementation Notes"..
which says
G.1 Network Guidelines
The HL7 2.6 standard does not define a network communications protocol. Beginning with HL7 2.2, the definitions of lower layer protocols were moved to the Implementation Guide, but are not HL7 requirements. The IHE Framework makes these recommendations:
Applications shall use the Minimal Lower Layer Protocol defined in Appendix C of the HL7 Implementation Guide.
An application that wants to send a message (initiate a transaction) will initiate a network connection to start the transaction. The receiver application will respond with an acknowledgement or response to query but will not initiate new transactions on this network connection
G.1.1 Acknowledgment Modes
ACKNOWLEDGMENT MESSAGES
Acknowledgment messages may be defined on an application basis. However the simple general acknowledgment message (ACK) may be used where the application does not define a special message (application level acknowledgment) and in other cases as described in Section 2.9, "Message Processing Rules".
The IHE PCD transaction PCD-03 supports „enhanced mode‟ acknowledgements. See discussion under PCD-03 Transactions as well as in B.1 MSH – Message Header Segment and B.2 MSA – Message Acknowledgement Segment
and document "Health Level Seven, Version 2.6 © 2007, Chapter 2: Control" coming from the "HL7 Messaging Standard Version 2.6" package which can be downloaded from http://www.hl7.org/implement/standards/product_brief.cfm?product_id=185 describes the accept and validate behavior in
2.9.2 Message response using the original processing rules
..too long to quote..
2.9.3 Response using enhanced acknowledgement
..too long to quote..
depending on the values of MSH-15 Accept Acknowledgement Type and MSH-16 Application Acknowledgment Type fields in the HL7 message
The above chapters from the HL7 standard contain what you want to read and implement/support.
EDIT:
Simply put, in HL7 protocol in every message sent the sender may request an ACK receipt by flagging appropriate fields in the message header segment. IHE does not remove this rule and does not enforce any other but enables any other convention to be defined on an application basis. Correct expected behavior is defined by the HL7 specification and in order to get it right and create a conforming implementation (without hidden surprises for your 3rd parties) you may need to read it several times (see also Stack Overflow: How can I make my system HL7 certified?)
For example this is how HAPI library handles the ACKing, snippet comes from http://sourceforge.net/p/hl7api/code/764/tree/tags/Root_REL_1_2/hapi-mvn/hapi-base/src/main/java/ca/uhn/hl7v2/protocol/impl/ProcessorImpl.java
/**
* #see ca.uhn.hl7v2.protocol.Processor#cycle(boolean)
*/
public void cycle(boolean expectingAck) throws HL7Exception {
log.debug("In cycle({})", expectingAck);
cleanReservations();
cleanAcceptAcks();
cleanReservedMessages();
Transportable in = null;
try {
if (expectingAck) {
in = tryReceive(myContext.getLocallyDrivenTransportLayer());
} else {
in = tryReceive(myContext.getRemotelyDrivenTransportLayer());
}
} catch (TransportException e) {
try {
Thread.sleep(1000);
} catch (InterruptedException e1) {}
throw e;
}
// log
if (in != null) {
log.debug("Received message: {}", in.getMessage());
} else {
log.debug("Received no message");
}
// If we have a message, handle it
if (in != null) {
String acceptAckNeeded = null;
// String appAckNeeded = null;
String ackCode = null;
String ackId = null;
try {
String[] fieldPaths = {"MSH-15", "MSH-16", "MSA-1", "MSA-2"};
String[] fields = PreParser.getFields(in.getMessage(), fieldPaths);
acceptAckNeeded = fields[0];
// appAckNeeded = fields[1];
ackCode = fields[2];
ackId = fields[3];
} catch (HL7Exception e) {
log.warn("Failed to parse accept ack fields in incoming message", e);
}
if (ackId != null && ackCode != null && ackCode.startsWith("C")) {
long expiryTime = System.currentTimeMillis() + 1000 * 60;
myAcceptAcks.put(ackId, new ExpiringTransportable(in, expiryTime));
} else {
AcceptAcknowledger.AcceptACK ack = AcceptAcknowledger.validate(getContext(), in);
if ((acceptAckNeeded != null && acceptAckNeeded.equals(AL))
|| (acceptAckNeeded != null && acceptAckNeeded.equals(ER) && !ack.isAcceptable())
|| (acceptAckNeeded != null && acceptAckNeeded.equals(SU) && ack.isAcceptable())) {
trySend(myContext.getRemotelyDrivenTransportLayer(), ack.getMessage());
}
if (ack.isAcceptable()) {
if (isReserved(ackId)) {
log.debug("Received expected ACK message with ACK ID: {}", ackId);
removeReservation(ackId);
long expiryTime = System.currentTimeMillis() + 1000 * 60 * 5;
myAvailableMessages.put(ackId, new ExpiringTransportable(in, expiryTime));
} else {
log.debug("Sending message to router");
Transportable out = myContext.getRouter().processMessage(in);
sendAppResponse(out);
}
} else {
// TODO: should we do something more here? Might be nice to
// allow a configurable handler for this situation
log.warn("Incoming message was not acceptable");
}
}
} else {
String transport = expectingAck ? " Locally driven " : "Remotely driven";
log.debug("{} TransportLayer.receive() returned null.", transport);
}
sleepIfNeeded();
log.debug("Exiting cycle()");
}
Thanks for your answer :)
of course that it is better to use an ACK to make sure if the receiver is getting the message but what I wanted to know if it was mandatory or not using the PCD-01 transaction.
I've read your documents and what I've understood is that the use of ACK depends on the MSH-15 and MSH-16 fields content, but with the following information:
An application that wants to send a message (initiate a transaction) will initiate a network connection to start the transaction. The receiver application will respond with an acknowledgement or response to query but will not initiate new transactions on this network connection
I understand that the ACK is only at the beginning of the connection not after every message, is it right?
Background
I am developing an Android App which provides a simple HTTP/HTTPS server. If the HTTPS serving is configured then on every connection an increasing native memory usage is observed which eventually leads to an app crash (oom), while using the HTTP configuration keeps the native memory usage relative constant. The app's Java VM keeps relative constant in both configurations.
The app serves an HTML page which contains a javascript with periodic polling (one json poll every second), so calling the app page using the HTTPS configuration and keeping the page open for several hours will lead to the mentioned out-of-memory because of increasing native memory usage. I have tested many SSLServerSocket and SSLContext configurations found on internet with no luck.
I observe the same problem on various Android devices and various Android versions beginning with 2.2 up to 4.3.
The code for handling client requests is the same for both configurations HTTP/HTTPS. The only difference between the two configurations is the setup of the server socket. While in the case of HTTP server socket one single line similar to this "ServerSocket serversocket = new ServerSocket(myport);" does the job, in the case of HTTPS server setup the usual steps for setting up the SSLContext are taken -- i.e. setting up the keymanager and initializing the SSLContext. For now, I use the default TrustManager.
Need For Your Advice
Does somebody know about any memory leak problems in Android's default TLS Provider using OpenSSL? Is there something special I should consider to avoid the leak in the native memory? Any hint is highly appreciated.
Update: I have also tried both TLS providers: OpenSSL and JSSE by explicitly giving the provider name in SSLContext.getInstance( "TLS", providerName ). But that did not change anything.
Here is a code block which demonstrates the problem. Just create a sample app put it into the bottom of the main activity's onCreate and build & run the app. Make sure that your Wifi is on and call the HTML page by following address:
https://android device IP:9090
Then watch the adb logs, after a while you will see the native memory beginning to increase.
new Thread(new Runnable() {
public void run() {
final int PORT = 9090;
SSLContext sslContext = SSLContext.getInstance( "TLS" ); // JSSE and OpenSSL providers behave the same way
KeyManagerFactory kmf = KeyManagerFactory.getInstance( KeyManagerFactory.getDefaultAlgorithm() );
KeyStore ks = KeyStore.getInstance( KeyStore.getDefaultType() );
char[] password = KEYSTORE_PW.toCharArray();
// we assume the keystore is in the app assets
InputStream sslKeyStore = getApplicationContext().getResources().openRawResource( R.raw.keystore );
ks.load( sslKeyStore, null );
sslKeyStore.close();
kmf.init( ks, password );
sslContext.init( kmf.getKeyManagers(), null, new SecureRandom() );
ServerSocketFactory ssf = sslContext.getServerSocketFactory();
sslContext.getServerSessionContext().setSessionTimeout(5);
try {
SSLServerSocket serversocket = ( SSLServerSocket )ssf.createServerSocket(PORT);
// alternatively, the plain server socket can be created here
//ServerSocket serversocket = new ServerSocket(9090);
serversocket.setReceiveBufferSize( 8192 );
int num = 0;
long lastnatmem = 0, natmemtotalincrease = 0;
while (true) {
try {
Socket soc = (Socket) serversocket.accept();
Log.i(TAG, "client connected (" + num++ + ")");
soc.setSoTimeout(2000);
try {
SSLSession session = ((SSLSocket)soc).getSession();
boolean valid = session.isValid();
Log.d(TAG, "session valid: " + valid);
OutputStream os = null;
InputStream is = null;
try {
os = soc.getOutputStream();
// just read the complete request from client
is = soc.getInputStream();
int c = 0;
String itext = "";
while ( (c = is.read() ) > 0 ) {
itext += (char)c;
if (itext.contains("\r\n\r\n")) // end of request detection
break;
}
//Log.e(TAG, " req: " + itext);
} catch (SocketTimeoutException e) {
// this can occasionally happen (handshake timeout)
Log.d(TAG, "socket timeout: " + e.getMessage());
if (os != null)
os.close();
if (is != null)
is.close();
soc.close();
continue;
}
long natmem = Debug.getNativeHeapSize();
long diff = 0;
if (lastnatmem != 0) {
diff = natmem - lastnatmem;
natmemtotalincrease += diff;
}
lastnatmem = natmem;
Log.i(TAG, " answer the request, native memory in use: " + natmem / 1024 + ", diff: " + diff / 1024 + ", total increase: " + natmemtotalincrease / 1024);
String html = "<!DOCTYPE html><html><head>";
html += "<script type='text/javascript'>";
html += "function poll() { request(); window.setTimeout(poll, 1000);}\n";
html += "function request() { var xmlHttp = new XMLHttpRequest(); xmlHttp.open( \"GET\", \"/\", false ); xmlHttp.send( null ); return xmlHttp.responseText; }";
html += "</script>";
html += "</head><body onload=\"poll()\"><p>Refresh the site to see the inreasing native memory when using HTTPS: " + natmem + " </p></body></html> ";
byte[] buffer = html.getBytes("UTF-8");
PrintWriter pw = new PrintWriter( os );
pw.print("HTTP/1.0 200 OK \r\n");
pw.print("Content-Type: text/html\r\n");
pw.print("Content-Length: " + buffer.length + "\r\n");
pw.print("\r\n");
pw.flush();
os.write(buffer);
os.flush();
os.close();
} catch (IOException e) {
e.printStackTrace();
}
soc.close();
}
catch (IOException e) {
e.printStackTrace();
}
}
} catch (SocketException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}).start();
-- EDIT --
I have uploaded a sample app project called SSLTest for eClipse which demonstrates the problem:
http://code.google.com/p/android/issues/detail?id=59536
-- UPDATE --
Good news: today the reported Android issue above was identified and proper submissions were made to fix the memory leak. For more details see the link above.
I imagine this would be a substantial time investment, but I see that Valgrind has been ported to Android. You could try getting that up and running. Of course, if you find there's an internal memory leak, there isn't a lot you can do about it except attempt to get the bug fixed in future Android releases.
As a workaround, you could make your application multi-process and put the https service in a separate process. That way you could restart it periodically, avoiding OOM. You might also have to have a third process just accepting port 443 connections and passing them on to the https worker - in order to avoid tiny outages when the https worker is restarted.
This also sounds like a substantial time investment :) But it would presumably successfully avoid the problem.
--- EDIT: More detail ---
Yes, if you have a main application with its own UI, a worker process for handling SSL and a worker process for accepting the SSL requests (which as you say probably can't be 443), then on top of your normal Activity classes, you would have two Service classes, and the manifest would place them in separate processes.
Handling SSL process: Rather than waiting for an OOM to crash the service, the service could monitor its own Debug.getNativeHeapSize(), and explicitly restart the service when it increased too much. Either that, or restart automatically after every 100 requests or so.
Handling listening socket process: This service would just listen on the TCP port you choose and pass on the raw data to the SSL process. This bit needs some thought, but the most obvious solution is to just have the SSL process listen on a different local port X (or switch between a selection of different ports), and the listening socket process would forward data to port X. The reason for having the listening socket process is to gracefully handle the possibility that X is down - as it might be whenever you restart it.
If your requirements allow for there being occasional mini-outages I would just do the handling SSL process, and skip the listening socket process, it's a relatively simple solution then - not that different to what you'd do normally. It's the listening socket process that adds complexity to the solution...
Does it help to explicitly close the input stream? In the sample code the input stream seems to only be closed in the case of a SocketTimeoutException exception.
--EDIT--
You could rename run() to run2() and move the while loop into run() and remove it from run2() and see if that makes a difference? This couldn't be a solution but would tell you if any of the long-lived objects free up the memory when their references are dropped.
There is one detail I would recommend changing in your implementation.
Make a list of all your resource variables, for example Sockets, Streams, Writers, etc. Be sure to have the declaration outside your try statement and be sure to do cleanup / closing in the finally statement. I normally do something like this to be 100% sure:
InputStream in = null;
OutputStream out = null;
try {
//assign a proper value to in and out, and use them as needed.
} catch(IOException e) {
//normal error handling
} finally {
try {
in.close();
} catch(IOException e) {}
try {
out.close();
} catch(IOException e) {}
}
It looks a little bit confusing, but imagine you use your in Stream inside the try block and you get some Exception, then your Streams never get closed and that is a potential reason for memory leaks.
I cannot guarantee that this is the reason, but it should be a good startup point.
About managing your service. I had a lot of bad experiences with Android services because I was running them in the same thread as the GUI. Under some circumstances, Android will see some code that is executing for too long and kill your main process in order to protect from crashes. The solution I found was to follow the suggestion from this tutorial (look at point 4):
http://www.vogella.com/articles/AndroidServices/article.html
After this, my service just worked as expected and didn't interfere with my GUI Process.
Regards
I've been asking questions regarding my Android project that continually plots Bluetooth data in real-time.
Basically what I've already done is create a first version of my app by cobbling together some open source code Blueterm and OrientationSensorExample
It's been suggested that I add a thread, a handler, a Service, or use Async Task, or AIDL, etc. But I don't know how to use any of these and would appreciate an explanation.
Here's a description of the Blueterm open source code I started with (see link above). Blueterm is basically a terminal emulator program that communicates over Bluetooth. It consists of several activities with Blueterm being the most important. It discovers, pairs, and connects with a remote Bluetooth device that supports SPP/RfComm. When connected I can use Blueterm to configure the remote device by sending it commands to turn on sampling, change the number of channels to sample (to one channel), change to format of the incoming data (I like comma separated data), etc
Here's a description of the OrientationSensorExample open source code I started with (see link above). It's basically an example application of the AnroidPlot library. The OrientationSensor activity implements SensorEventListener. This includes overriding onSenorChanged() which is called whenever new orientation sensor data is taken, and it redraws the graph.
Having cobbled together these two open source projects (Blueterm and OrientationSensorExample) into one application (Blueterm) here's a description of how the overall application (Blueterm) works. When I start Blueterm the whole screen emulates a nice blue terminal. From the Options Menu I discover, pair with, connect to, and configure a remote bluetooth device as described above. Once I have configured the remote device, I go again to the Options Menu and select "Plot data" which launches the Plot activity. The terminal emulator goes away, and a nice scrolling real-time plot from the Plot activity shows up.
As far as I can tell there is a background thread that calls an update() method as follows:
/**
* Look for new input from the ptty, send it to the terminal emulator.
*/
private void update() {
int bytesAvailable = mByteQueue.getBytesAvailable();
int bytesToRead = Math.min(bytesAvailable, mReceiveBuffer.length);
try {
int bytesRead = mByteQueue.read(mReceiveBuffer, 0, bytesToRead);
append(mReceiveBuffer, 0, bytesRead);
//VTR use existing handler that calls update() to get data into plotting activity
Plot.plotData(mReceiveBuffer, 0, bytesRead);
} catch (InterruptedException e) {
//VTR OMG their swallowing this exception
}
}
In the update() method I found it convenient to call my Plot.plotData() method and pass it the same date that is passed to the append() method to plot the data. NOTE: This only works if plotData() is a static method. No one has been able to explain why.
Anyway plotData() is a static method and here's how it and it's helper methods look now:
private static StringBuffer strData = new StringBuffer("");
public static void plotData(byte[] buffer, int base, int length) {
Log.i("Entering: ", "plotData()");
/*
byte[] buffer = (byte[]) msg.obj;
int base = msg.arg1;
int length = msg.arg2;
*/
for (int i = 0; i < length; i++) {
byte b = buffer[base + i];
try {
if (true) {
char printableB = (char) b;
if (b < 32 || b > 126) {
printableB = ' ';
}
Log.w("Log_plotData", "'" + Character.toString(printableB)
+ "' (" + Integer.toString(b) + ")");
strData.append(Character.toString(printableB));
if (b == 10)
{
Log.i("End of line: ", "processBlueData()");
Log.i("strData", strData.toString());
splitData(strData);
strData = new StringBuffer("");
}
}
} catch (Exception e) {
Log.e("Log_plotData_exception", "Exception while processing character "
+ Integer.toString(i) + " code "
+ Integer.toString(b), e);
}
}
Log.i("Leaving: ", "plotData()");
}
private static void splitData(StringBuffer strBuf) {
String strDash = strBuf.toString().trim();
String[] strDashSplit = strDash.split("-");
for (int ndx = 0; ndx < strDashSplit.length; ndx++)
{
if (strDashSplit[ndx].length() > 0)
Log.i("strDashSplit", ndx + ":" + strDashSplit[ndx]);
String strComma = strDashSplit[ndx].trim();
String[] strCommaSplit = strComma.split(",");
for (int mdx = 0; mdx < strCommaSplit.length; mdx++)
{
if (strCommaSplit[mdx].length() > 0)
Log.i("strCommaSplit", mdx + ":" + strCommaSplit[mdx]);
if (mdx == 1)
{
int raw = Integer.parseInt(strCommaSplit[1],16);
Log.i("raw", Integer.toString(raw));
float rawFloat = raw;
Log.i("rawFloat", Float.toString(rawFloat));
float ratio = (float) (rawFloat/65535.0);
Log.i("ratio", Float.toString(ratio));
float voltage = (float) (5.0*ratio);
Log.i("voltage", Float.toString(voltage));
nowPlotData(voltage);
}
}
}
}
public static void nowPlotData(float data) {
// get rid the oldest sample in history:
if (plotHistory.size() > HISTORY_SIZE) {
plotHistory.removeFirst();
}
// add the latest history sample:
plotHistory.addLast(data);
// update the plot with the updated history Lists:
plotHistorySeries.setModel(plotHistory, SimpleXYSeries.ArrayFormat.Y_VALS_ONLY);
//VTR null pointer exception?
if (plotHistoryPlot == null)
Log.i("aprHistoryPlot", "null pointer exception");
// redraw the Plots:
plotHistoryPlot.redraw();
}
If it is strongly recommended that plotData() not be a static method and that I should do something else please explain here and how. Thanks!
This might be a question much better suited for Code Review, rather than here. Perhaps you can reformulate to post it there, or trim it a lot to repost it here.
Furthermore, to answer: "It's been suggested that I add a thread, a handler, a Service, or use Async Task, or AIDL, etc. But I don't know how to use any of these and would appreciate an explanation.", the best advise would be to link you to a book about android, such as: http://commonsware.com/Android/ . Chapters 35 and 36 deal with services, while chapter 20 is about threads. You will never get an answer as complete as those chapters here.