Develop Reference

Mock HttpResponse with Robolectric

Using Robolectric 2.3-SNAPSHOT, I want to test an object that'll execute a request in the background. In order to isolate it, I'm trying to mock the HttpResponse returned, without success after some hours invested.
I've created a project that anyone can clone. Simly run ./gradlew check https://github.com/Maragues/RobolectricDummyProject (git clone https://github.com/Maragues/RobolectricDummyProject.git)
I've tried
Robolectric.setDefaultHttpResponse(200, "my_mocked_word");
MockWebServer (https://code.google.com/p/mockwebserver/)
But the tests fail because they query the real URL
private static final String MOCKED_WORD = "MOCKED";
#Test
public void mockedRequestUsingMockServer() throws Exception {
mMockWebServer.enqueue(new MockResponse().setBody(MOCKED_WORD));
mMockWebServer.play();
Robolectric.getFakeHttpLayer().interceptHttpRequests(false);
Robolectric.getFakeHttpLayer().interceptResponseContent(false);
String result = request.loadDataFromNetwork();
assertEquals(MOCKED_WORD, result);
}
#Test
public void mockedRequestUsingRobolectric() throws Exception {
Robolectric.setDefaultHttpResponse(200, MOCKED_WORD);
String result = request.loadDataFromNetwork();
assertEquals(MOCKED_WORD, result);
}
The code executing the request
public String loadDataFromNetwork() throws Exception {
// With Uri.Builder class we can build our url is a safe manner
Uri.Builder uriBuilder = Uri.parse("http://robospice-sample.appspot.com/reverse").buildUpon();
uriBuilder.appendQueryParameter("word", word);
String url = uriBuilder.build().toString();
HttpURLConnection urlConnection = (HttpURLConnection) new URL(url)
.openConnection();
String result = IOUtils.toString(urlConnection.getInputStream());
urlConnection.disconnect();
return result;
}
Possibly related questions
Can't capture HTTP request with robolectric (I've tried that without success. Perhaps I'm missing something)
Anyone had success mocking HttpRequests with Robolectric? (I'm not using eclipse)

You're disabling Roboelectric's HTTP layer, so you're using the real HTTP layer. This means that there's no clever magic happening under the hood of your test: when you send an HTTP request, it's really going out onto the internet (as you are seeing).
MockWebServer doesn't stop this. It just sets up a web server locally, that your test can connect to.
So to resolve this problem, you need to stop attempting to connect to a real server, and instead, connect to the mock server. To do this, yo need to inject/set the URL in the request.
#Test
public void mockedRequestUsingMockServer() throws Exception {
mMockWebServer = new MockWebServer();
mMockWebServer.play();
mMockWebServer.enqueue(new MockResponse().setResponseCode(200).setBody(MOCKED_WORD));
request.myUrl = mMockWebServer.getUrl("/");
String result = request.loadDataFromNetwork();
assertEquals(MOCKED_WORD, result);
mMockWebServer.shutdown();
}

It turns out that Robolectric's FakeHttpLayer only works with Apache's HttpClient, which is highly discouraged on versions greater than Froyo. Extracted from Robolectric's Google Group
That being said, the usage of HttpUrlConnection will cause you trouble. I'd try to use Android's implementation of HttpClient where possible, since Robolectric intercepts all calls to that library and lets you set up canned responses to your HTTP calls. We're looking at doing the same for HttpUrlConnection, though it's not clear when that'll happen.
Apart from that, a unit test should not need to mock the HTTP layer. My approach was wrong from the beginning.

You can try this(ref:https://github.com/square/okhttp/tree/master/mockwebserver).
// Create a MockWebServer. These are lean enough that you can create a new
// instance for every unit test.
MockWebServer server = new MockWebServer();
// Schedule some responses.
server.enqueue(new MockResponse().setBody("it's all cool"));
// Start the server.
server.play();
// Ask the server for its URL. You'll need this to make HTTP requests.
//Http is my own http executor.
Http.Response response = http.get(server.getUrl("/"));
then, you can compare the response to server.enqueue(new MockResponse().setBody("it's all cool"));
MockWebServer is a part of okhttp https://github.com/square/okhttp/tree/master/mockwebserver. the URLConnectionImpl in android 4.4 have been changed from defaultHttpClient to Okhttp.

Android HttpUrlConnection EOFException

I would like to know if there are known issues on Android with HttpUrlConnection and POST requests. We are experiencing intermittent EOFExceptions when making POST requests from an Android client. Retrying the same request will eventually work. Here is a sample stack trace:
java.io.EOFException
at libcore.io.Streams.readAsciiLine(Streams.java:203)
at libcore.net.http.HttpEngine.readResponseHeaders(HttpEngine.java:579)
at libcore.net.http.HttpEngine.readResponse(HttpEngine.java:827)
at libcore.net.http.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:283)
at libcore.net.http.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:497)
at libcore.net.http.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:134)
There are many similar bug reports and posts to stack overflow but I cannot understand if there really is an issue and if so, what versions of Android are affected and what the proposed fix/work around is.
Here are some of the similar reports I am referring to:
Android HttpsUrlConnection eofexception
Android HttpURLConnection throwing EOFException
EOFException and FileNotFoundException in HttpURLConnection getInputStream()
https://code.google.com/p/google-http-java-client/issues/detail?id=213
https://code.google.com/p/android/issues/detail?id=29509
https://code.google.com/p/google-http-java-client/issues/detail?id=230
https://code.google.com/p/android/issues/detail?id=41576
Here is a potential Android framework fix
https://android.googlesource.com/platform/libcore/+/19aa40c81c48ff98ccc7272f2a3c41479b806376
I do know there was an issue with poisoned connections in the connection pool in pre-Froyo but these issues are occurring on new ICS+ devices exclusively. If there were a problem on later devices I would expect some kind of official Android documentation of the issue.

Our conclusion is that there is an issue in the Android platform. Our workaround was to catch the EOFException and retry the request N number of times. Below is the pseudo code:
private static final int MAX_RETRIES = 3;
private ResponseType fetchResult(RequestType request) {
return fetchResult(request, 0);
}
private ResponseType fetchResult(RequestType request, int reentryCount) {
try {
// attempt to execute request
} catch (EOFException e) {
if (reentryCount < MAX_RETRIES) {
fetchResult(request, reentryCount + 1);
}
}
// continue processing response
}

HttpURLConnection library internally maintains a pool of Connections. So, whenever a request is send, it first checks if there is an existing connection already present in the pool, based on which it decides to create a new one.
These connections are nothing but sockets, and this library by default does not closes these sockets. It may sometimes happen that a connection (socket) which is not currently being used and is present in the pool is no longer usable as the Server may choose to terminate the connection after some time. Now, since the connection even though is closed by the server, the library does not knows about it and assumes the connection/socket to be still connected. Thus it sends the new request using this stale connection and hence we get EOFException.
The best way to handle this is to check the Response Headers after each request you send. The server always sends a "Connection: Close" before terminating a connection (HTTP 1.1). So, you can use getHeaderField() and check for "Connection" field. Another thing to note is that server ONLY sends this connection field when it is about to terminate the connection. So, you need to code around this with the possibility of getting a "null" in the normal case (when server is not closing the connection)

This workaround tends to be reliable and performant:
static final int MAX_CONNECTIONS = 5;
T send(..., int failures) throws IOException {
HttpURLConnection connection = null;
try {
// initialize connection...
if (failures > 0 && failures <= MAX_CONNECTIONS) {
connection.setRequestProperty("Connection", "close");
}
// return response (T) from connection...
} catch (EOFException e) {
if (failures <= MAX_CONNECTIONS) {
disconnect(connection);
connection = null;
return send(..., failures + 1);
}
throw e;
} finally {
disconnect(connection);
}
}
void disconnect(HttpURLConnection connection) {
if (connection != null) {
connection.disconnect();
}
}
This implementation relies on the fact that the default number of connections that can be opened with a server is 5 (Froyo - KitKat). This means that up to 5 stale connections may exist, each of which will have to be closed.
After each failed attempt, the Connection:close request property will cause the underlying HTTP engine to close the socket when connection.disconnect() is called. By retrying up to 6 times (max connections + 1), we ensure that the last attempt will always be given a new socket.
The request may experience additional latency if no connections are alive, but that is certainly better than an EOFException. In that case, the final send attempt won't immediately close the freshly opened connection. That's the only practical optimization that can be made.
Instead of relying on the magic default value of 5, you may be able to configure the system property yourself. Keep in mind that this property is accessed by a static initializer block in KitKat's ConnectionPool.java, and it works like this in older Android versions too. As a result, the property may be used before you have a chance to set it.
static final int MAX_CONNECTIONS = 5;
static {
System.setProperty("http.maxConnections", String.valueOf(MAX_CONNECTIONS));
}

Yes. There is a problem in the Android platform, specifically, in Android libcore with version 4.1-4.3.
The problem is introduced in this commit: https://android.googlesource.com/platform/libcore/+/b2b02ac6cd42a69463fd172531aa1f9b9bb887a8
Android 4.4 switched http lib to "okhttp" which doesn't have this problem.
Problem explained as follow:
On android 4.1-4.3, when you are using URLConnection/HttpURLConnection to POST with "ChunkedStreamingMode" or "FixedLengthStreamingMode" set, URLConnection/HttpURLConnection will not do silent retry if the reused connection is stale. You should retry POST at most "http.maxConnections+1" times in your code, just as previous answers suggest.

I suspect it might be the server that is at fault here, and the HttpURLConnection is not as forgiving as other implementations. That was the cause of my EOFException. I suspect in my case this would not be intermittent (fixed it before testing the N retry workaround), so the answers above relate to other issues and be a correct solution in those cases.
My server was using python SimpleHTTPServer and I was wrongly assuming all I needed to do to indicate success was the following:
self.send_response(200)
That sends the initial response header line, a server and a date header, but leaves the stream in the state where you are able to send additional headers too. HTTP requires an additional new line after headers to indicate they are finished. It appears if this new line isn't present when you attempt to get the result body InputStream or response code etc with HttpURLConnection then it throws the EOFException (which is actually reasonable, thinking about it). Some HTTP clients did accept the short response and reported the success result code which lead to me perhaps unfairly pointing the finger at HttpURLConnection.
I changed my server to do this instead:
self.send_response(200)
self.send_header("Content-Length", "0")
self.end_headers()
No more EOFException with that code.

This worked for me.
public ResponseObject sendPOST(String urlPrefix, JSONObject payload) throws JSONException {
String line;
StringBuffer jsonString = new StringBuffer();
ResponseObject response = new ResponseObject();
try {
URL url = new URL(POST_URL);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setReadTimeout(10000);
connection.setConnectTimeout(15000);
connection.setRequestMethod("POST");
connection.setRequestProperty("Accept", "application/json");
connection.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
OutputStream os = connection.getOutputStream();
os.write(payload.toString().getBytes("UTF-8"));
os.close();
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
while ((line = br.readLine()) != null) {
jsonString.append(line);
}
response.setResponseMessage(connection.getResponseMessage());
response.setResponseReturnCode(connection.getResponseCode());
br.close();
connection.disconnect();
} catch (Exception e) {
Log.w("Exception ",e);
return response;
}
String json = jsonString.toString();
response.setResponseJsonString(json);
return response;
}

connection.addRequestProperty("Accept-Encoding", "gzip");
is the answer

Get request error in resting framework

I am creating an android client which connects to the web service and gets a response code. The problem is the android app crashes whenever I run the following code:
RequestParams params = new BasicRequestParams();
params.add("client_id", "24f8b46fc9db409012830ca264ad7bcf");
params.add("response_type", "code");
ServiceResponse response=Resting.get("http://pricewatch.ap01.aws.af.cm/api/pricewatch/oAuth/auth",80,params);
IContentData contentData = response.getContentData();
String content = (String) contentData.getContent();
JSONObject jsonObject = new JSONObject(content);
String code = (String) jsonObject.get("code");
return code;
It generates a java.lang.NullPointerException with these line:
IContentData contentData = response.getContentData();
Could someone help with these? I've been stuck with these. Thanks in advance!

I run this code many times in the emulator but never worked. However, when I run this in an android device it worked! NullPointerException happens when it doesn't get anything or any response in the server. But what I don't understand is why does this happen? Is there a difference between an emulator and an android device in terms of executing an android application?

JDOM Throwing Parse Exceptions with bad ascii characters

I'm using JDOM with my Android project, and every time I get a certain set of characters in my server response, I end up with these error messages:
05-04 10:08:46.277: E/PARSE: org.jdom.input.JDOMParseException: Error on line 95 of document UTF-8: At line 95, column 5263: unclosed token
05-04 10:08:46.277: E/Error Handler: Handler failed: org.jdom.input.JDOMParseException: Error on line 1: At line 1, column 0: syntax error
When I make the same query through google chrome, I can see that all of the XML came through fine, and that there are in fact no areas where a token is not closed. I have run into this problem several times throughout the development of the application, and the solution has always been to remove odd ascii characters (copyright logos, or trademark characters, etc. that got copied/pasted into those data fields). How can I get it to either a remove those characters, or b strip them and continue the function. Here's an example of one of my parse functions.
public static boolean parseUserData(BufferedReader br) {
SAXBuilder builder = new SAXBuilder();
Document document = null;
try {
document = builder.build(br);
/* XML Output to Logcat */
if (document != null) {
XMLOutputter outputter = new XMLOutputter(
Format.getPrettyFormat());
String xmlString = outputter.outputString(document);
Log.e("XML", xmlString);
}
Element rootNode = document.getRootElement();
if (!rootNode.getChildren().isEmpty()) {
// Do stuff
return true;
}
} catch (Exception e) {
GlobalsUtil.errorUtil
.setErrorMessage("Error Parsing XML: User Data");
Log.e(DEBUG_TAG, e.toString());
return false;
}
}

It distinctly sounds like a character encoding issue. I think duffymo is correct in his assessment. I have two comments though ....
If you are getting your data through a URL you should be using the URLConnection.getContentType() to get the charset (if it is set and the charset is not null) to set up the InputStreamReader on the URL's InputStream...
Have you tried JDOM 2.0.1? It is the first JDOM version that is fully tested on Android... (and the only 'supported' JDOM version on Android). JDOM 2.0.1 also has a number of performance tweaks, and memory optimizations that should make your processing faster. It also fixes a number of bugs.... though from what I see you should not run in to any bug problems.....
Check out https://github.com/hunterhacker/jdom/wiki/JDOM2-Migration-Issues and https://github.com/hunterhacker/jdom/wiki/JDOM2-and-Android

Is the BufferedReader constructed to take the encoding argument? Perhaps you need to tell the Reader or InputStream that you pass to use UTF-8.

Stream android logcat output to an sd card

I want to achieve the following but so far, no luck
Open a file in the SD Card when the android application first started.
Stream the logcat output to the file.
When the application exits, stop the logcat streaming.
In my ApplicationClass extends Application onCreate() method, I do this
wwLogManager.openLogFile();
and here's the code in the openLogFile()
private static final String LOGCAT_COMMAND = "logcat -v time -f ";
String timestamp = Long.toString(System.currentTimeMillis());
String logFileName = BuildConstants.LOG_LOCATION + "_" + timestamp + ".log";
mLogFile = new File(Environment.getExternalStorageDirectory() + logFileName);
mLogFile.createNewFile();
String cmd = LOGCAT_COMMAND + mLogFile.getAbsolutePath();
Runtime.getRuntime().exec(cmd);
I do get log files in the sd card, but the log output in these files do not have any trace of the Log.i() calls that I placed in my activities. Is the logcat command that I used here correct? Thanks!

I apologize if I am misunderstanding your goals, but perhaps you could use the java.util.logging API instead of using Logcat or the Android Logging mechanism.
Like the Android logging API, the java.util.logging API allows you to easily log messages at various levels, such as FINE, FINER, WARN, SEVERE, etc.
But the standard logging API has additional advantages, too. For example, you can easily create a log file by using a FileHandler. In fact, FileHandler has a built-in log rotation mechanism, so you don't have to worry (so much) about cleaning up the log files. You can also create a hierarchy of Loggers; so, for example, if you have two Loggers, com.example.foo and com.example.foo.bar, changing the logging level of the former will also change the logging level of the latter. This will even work if the two Loggers are created in different classes! Moreover, you change logging behavior at runtime by specifying a logging configuration file. Finally, you can customize the format of the log by implementing your own Formatter (or just use the SimpleFormatter to avoid the default XML format).
To use the standard logging API, you might try something like this:
// Logger logger is an instance variable
// FileHandler logHandler is an instance variable
try {
String logDirectory =
Environment.getExternalStorageDirectory() + "/log_directory";
// the %g is the number of the current log in the rotation
String logFileName = logDirectory + "/logfile_base_name_%g.log";
// ...
// make sure that the log directory exists, or the next command will fail
//
// create a log file at the specified location that is capped 100kB. Keep up to 5 logs.
logHandler = new FileHandler(logFileName, 100 * 1024, 5);
// use a text-based format instead of the default XML-based format
logHandler.setFormatter(new SimpleFormatter());
// get the actual Logger
logger = Logger.getLogger("com.example.foo");
// Log to the file by associating the FileHandler with the log
logger.addHandler(logHandler);
}
catch (IOException ioe) {
// do something wise
}
// examples of using the logger
logger.finest("This message is only logged at the finest level (lowest/most-verbose level)");
logger.config("This is an config-level message (middle level)");
logger.severe("This is a severe-level message (highest/least-verbose level)");
The Android logging mechanism is certainly easy and convenient. It isn't very customizable, though, and log filtering must be done with tags, which can easily become unwieldy. By using the java.uitl.logging API, you can avoid dealing with a multitude of tags, yet easily limit the log file to specific parts of your application, gain greater control over the location and appearance of the log, and even customize logging behavior at runtime.

I repost my answer here so #JPM and others can see... The code basically just execute the logcat command and then build the log from the input stream.
final StringBuilder log = new StringBuilder();
try {
ArrayList<String> commandLine = new ArrayList<String>();
commandLine.add("logcat");
commandLine.add("-d");
ArrayList<String> arguments = ((params != null) && (params.length > 0)) ? params[0] : null;
if (null != arguments){
commandLine.addAll(arguments);
}
Process process = Runtime.getRuntime().exec(commandLine.toArray(new String[0]));
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = bufferedReader.readLine()) != null){
log.append(line);
log.append(LINE_SEPARATOR);
}
}
catch (IOException e){
//
}
return log;

Try manually setting a filter as described here:
http://developer.android.com/guide/developing/debugging/debugging-log.html#filteringOutput
Something like:
logcat ActivityManager:I MyApp:V *:S
If you replace "MyApp" with the log tags that you are using, that should show you all info (and greater) logs from ActivityManager, and all verbose (and greater) logs from your app.

I know this is a late answer to the question but I would highly recommend using Logback to write messages to a log file as well as the logcat.
Copy logback-android-1.0.10-2.jar and slf4j-api-1.7.5.jar to your libs folder.
Right-Click on both libs and from the menu select Build Path -> Add to Build Path.
Create a logback.xml file in your assets folder and enter the following:
%msg
WARN
${LOG_DIR}/log.txt
%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
To write a log:
public class MyActivity extends Activity
{
public static final Logger logger = LoggerFactory.getLogger(MyActivity.class);
protected void onCreate(Bundle b)
{
super(b);
try{
throw new Exception("Test");
}catch(Exception e){
logger.error("Something bad happened",e);
}
}
}