Develop Reference

Using Android Volley to perform a search on a website's search bar

So I've been dealing with my problems for some time now with the help of Internet, but I've only had minimal success.
The problem: I'm using Android with the Volley library to try and perform a search from two websites. The websites in question don't have an open API or anything to see how I could perform the searches. I've tried inspecting the HTTP requests in Firefox with the view opening with F12.
Website 1
Uses HTTPS protocol. I need to search from here using the Vnr number. From the resulting page I would parse the name(s) of the drugs.
Website 2
Uses HTTP protocol. From here I would search using the found drug name and I would get the drug description (in Finnish).
For Website 2 I tried the following code, which successfully returns the HTML file of the result page:
RequestQueue queue = Volley.newRequestQueue(this);
String url = "http://www.laakeinfo.fi/Medicine.aspx?m=758&d=2111337&i=ORION+PHARMA_RESILAR_RESILAR+oraaliliuos+3+mg/ml";
StringRequest request = new StringRequest(Request.Method.GET, url, new Response.Listener<String>() {
#Override
public void onResponse(String response) {
Log.d("HTTPTEST", "Response len: " + response.length());
writeToFile(response);
}
}, new Response.ErrorListener() {
#Override
public void onErrorResponse(VolleyError error) {
Log.d("HTTPTEST", "Error: " + error.getMessage());
}
}) {
#Override
protected Map<String, String> getParams() throws AuthFailureError {
Map<String, String> params = new HashMap<String, String>();
params.put("i", "Resilar");
return params;
}
#Override
public Map<String, String> getHeaders() throws AuthFailureError {
Map<String, String> params = new HashMap<String, String>();
params.put("Content-Type", "application/x-www-form-urlencoded");
return params;
}
};
queue.add(request);
Now this works because the URL points straight to where I want it to. If I changed the URL to "http://www.laakeinfo.fi/Search.aspx" and in getParams() I'd use instead params.put("Search1:txtSearch", "Resilar"); (the search bar's contents) it just returns the search page itself.
Now my questions is, is what I'm trying to do possible? I'm not too well in the know with web stuff, but it seems the pages hide the search logic and everything in a way that makes it difficult to perform the searches. If it is indeed possible, how could I go about it? Any hints would be appreciated in this attempt.

The quick answer is YES, you can do it! ... but it might be not that simple :)
So servers might check many many things in the request and even 'insignificant' data like Accept-Language: header might cause the service to return or not a response.
you have 2 main things you have to check and experiment with and 1 remark:
1) Headers
Host: xn--lkeinfo-5waa.fi
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Referer: http://xn--lkeinfo-5waa.fi/Search.aspx
Cookie: ASP.NET_SessionId=mxj22y2oofzwcnmtcen50c45
these might change the behavior so you need to experiment there. I would try to put all that I think might be useful or do not understand. Here I would remove just the 'Cookie' and 'Accept-Encoding'.
2) Params
__EVENTTARGET:"Search1$lbSubmit"
__EVENTARGUMENT:""
__VIEWSTATE:"/wEPDwUKMTc2NzY1NjI3Mg9kFgJmD2QWBAIDD2QWAgIFDxYCHgVjbGFzcwUGYWN0aXZlZAIFD2QWBGYPFgIfAAUJZnJtc2VhcmNoZAICDw8WBh4IQ3NzQ2xhc3MFA2J0bh4EVGV4dAUDSEFFHgRfIVNCAgJkZGQ2BqYJzINvRHYkGXr7+fwtg0L8cQ=="
__VIEWSTATEGENERATOR:"BBBC20B8"
__EVENTVALIDATION:"/wEWBALa8vLKCALQkcCMBAL43fjZDALY7fiNCoNRBovCyGbCM+mhYEACTRYyyeuF"
Search1:txtSearch:"Resilar"
Search1:hdLanguageId:"1"
Although 'Search1:txtSearch' seems the obvious one to make a search the other might play part in it to so I would add all of those and remove them 1 by one and try out.
3) (remark) response 302 this is a post request which does not return a response but rather redirects to one. You have to be careful enable debug logs and check what Volley returns as this response is considered as an error in volley. It might be handled by the HTTP Stack Volley uses but it depends which one is set. If this is the case this will be transparent to volley but otherwise some customization on Volley needs to be done or use another stack. For example android has adopted and baked OkHttp in Android source as the native client actually used by URLConnection. So earlier versions like KitKat would not handle this redirect. The new releases however handles it.

WebView unable to save cookie on Android 2.3.x

So I've been trying to get a WebView to work properly to authenticate a user session that is used in app. It works on all 4.0+ devices but when I try to use the same process on a 2.3.6 and 2.3.7 devices it throws this exception in the log:
com.myapp.WebActivity E/webkit﹕ parse cookie failed for: request_uri=xxxxxxxx; path=/; expires=Mon, 16-Jun-2014 00:00:00; domain=;
It doesn't crash the app but will not save this one cookie needed to properly authenticate the users session. I've looked into the Android Source code and it appears that when trying to save this specific cookie there is a RunTimeException that is happening. Here is the Android Source where the exception is thrown and caught. I'm not 100% sure where to look next since the cookies are saved properly on 4.0+ and it seems like a AOSP bug. Also, after the WebView loads the CookieManager doesn't have the cookie that it threw the exception on but has others.
So my real question is: Is there anyway to manually get a cookie returned from a WebView page load or can I get the cookie to be saved by the Android 2.3 WebView automatically somehow?
Thanks in advance.

Here is what I use to check and save a cookie that is picked up by a subsequent WebView. It works on Android 2.2 for sure, as well as Android 4.X.
Checking:
// see if the cookie is already set up
boolean cookieAlreadySetUp = false;
String cookie = cookieManager.getCookie(urlHost);
if ( cookie != null && cookie.length() > 0 ) {
String[] cookies = cookie.split(";");
for ( int i = 0; i < cookies.length; i++ ) {
String oneCookie = cookies[i];
if ( oneCookie.trim().equals(SharedData.COOKIE_STRING) ) {
cookieAlreadySetUp = true;
break;
}
}
}
Saving:
cookieManager.setAcceptCookie(true);
cookieManager.setCookie(urlHost, myCookieString);
CookieSyncManager.getInstance().sync();
// give time for the cookie to become known since the save process is asynchronous to this thread
SystemClock.sleep(100);
Be aware of the asynchronous nature of the cookie manager, both for adding or removing a cookie, as well as testing its existence.

Token authentication with Volley

If I have a server where I authenticate with username/password and get auth token for subsequent requests, what would be the best approach addressing this problem?
The flow should be like this:
- Start request
- If we don't have auth token - get it with username and password
- Make request with auth token
- If request failed because token expired, get new auth token with user name and password
- Retry request with new auth token
- Finish
I've noticed that Volley already might have something that might solve this issue - Authenticator https://android.googlesource.com/platform/frameworks/support/+/4474bc11f64b2b274ca6db5a1e23e8c1d143d5fa/volley/src/com/android/volley/toolbox/Authenticator.java It contains getAuthToken() and invalidateAuthToken() methods which would be exactly what I want. But it seems that it's never used in the library at all.

I used volley for an authentication system using longlive (LLT) and shortlive (SLT) tokens.
I did it manually but it really wasn't much work once you get it all laid out.
Have all secure requests subclass a baseSecureRequest that can handle this token mechanism common to all secure request in its onResponse() and onErrorResponse().
It becomes a little node.js style, where requests send other requests and await callbacks.
An app may have a dozen screens, with only half requiring auth access - so each screen should be ignorant as to the requirements of its request.
Scenario A
We attempt to send a secure request. We notice we don't have a SLT in
memory, so make a TokenRequest.
TokenRequest's onResponse() saves
that token to memory (let a singleton session manager hold onto it or
similar omni-present class)
Now callback to the original
concrete-class request object to continue with the newly updated token.
Scenario B
We send a secure request but our SLT is stale (expired)
The server returns an error code or msg that you can catch in the
general onErrorResponse() of your baseSecureRequest.
In this onError(), you send a refreshTokenRequest() object that
attempts to refresh the SLT in memory by requesting a new SLT from the server using the LLT.
the onResponse() of the refreshTokenRequest can now callback to the
original request to resend.
however the onErrorResponse() should probably abandon the entire thing because chances are
anything that isn't a connectivity error - is an error caused by
invalid LLT. If you keep trying to refresh with a bad LLT you will never get out.

You might want to use the AccountManager API in Android for authentication and authorization. You may follow the blog here.
For OAuth 2.0 server side implementation you may read the IETF draft V2-31 here.
For better understanding of OAuth 2.0 you may read the blog by Nitesh Kumar here.
For Server side implementation of OAuth 2.0 you can fork Apis Autherization Server repo at Github.
More implementation option can be found at the website of OAuth 2.0 here.

Did you see this blog post? https://smaspe.github.io/2013/06/06/volley-part2.html
Demonstrates a simple way of overriding request object to use twitter tokens.
#Override
public Map<String, String> getHeaders() throws AuthFailureError {
Map<String, String> headers = new HashMap<String, String>();
String auth = "Basic "
+ Base64.encodeToString((TwitterValues.CONSUMER_KEY
+ ":" + TwitterValues.CONSUMER_SECRET).getBytes(),
Base64.NO_WRAP);
headers.put("Authorization", auth);
return headers;
}

In my case, I changed a "Basic" authentication to a Token authentication as follows:
#Override
public Map<String, String> getHeaders() throws AuthFailureError {
Map<String,String> headers = new HashMap<>();
//todo Esta es una authenticación basica (usuario y contraseña)
/*String credentials = USER+":"+PASSWORD;
String auth = "Basic " + Base64.encodeToString(credentials.getBytes(), Base64.NO_WRAP);
headers.put("Authorization", auth);*/
//todo Esta es una authenticación con token (por tiempos)
headers.put("Authorization", "Bearer" + " " + "tokenString");//App.getToken()
return headers;
}
What I did was to save the Login in a global static variable and then be able to use it.

getToken() failed. Status BAD_AUTHENTICATION error
I also faced the same problem.
Solution: check whether the device is sign-in with your Google account.

Cookie corruption with multiple createHTTPClient Titanium calls

While creating an Android app in Appcelerator's Titanium that involves both webView and background calls, I ran into a problem / bug where the cookies were getting corrupted on multiple createHTTPClient calls.
Cookies were originally obtained from the webView:
var webview = Ti.UI.createWebView();
webview.url = 'http://www.example.com';
window.add(webview);
webview.addEventListener('load', function(e) {
cookies = e.source.evalJS("document.cookie");
Titanium.App.Properties.setString('cookies',cookies);
}
window.open({modal:true});
and then later used with a background call:
var loader = Titanium.Network.createHTTPClient();
loader.open("GET",base_url + url);
loader.onload = function() {
// process response
}
loader.setRequestHeader('Cookie',Titanium.App.Properties.getString("cookies"));
loader.send();
The first time the above createHTTPClient chunk of code was called, everything worked, but subsequent runs of the above code would send corrupted cookies. In Google App Engine (gae), printing out the request headers would look like this (broken):
logging.info('Request:\n%s' % self.request)
broken response (only the cookie portion of the request header is shown)
Cookie: auth="eyJfdXNlciI6WzYsMSwiVmRoZEtnYWZRMnNxazFjaVM0U1lKdCIsMTM1NzIyMzcwOSwxMzU3MjIzNzA5XX0\075|1357223709|4f622167f477a8c82cab196af4b0029af1a966d7", auth=eyJfdXNlciI6WzYsMSwiVmRoZEtnYWZRMnNxazFjaVM0U1lKdCIsMTM1NzIyMzcwOSwxMzU3MjIzNzA5XX0\075|1357225569|7a469fab7a38a437649c25620729e07c4607f617
Cookie2: $Version=1
working response
Cookie: auth="eyJfdXNlciI6WzYsMSwiVmRoZEtnYWZRMnNxazFjaVM0U1lKdCIsMTM1NzIyMzcwOSwxMzU3MjIzNzA5XX0\075|1357223709|4f622167f477a8c82cab196af4b0029af1a966d7"
...
I suspect the issue has something to do with unicode characters or something inside createHTTPClient. Two auth= statements are shown in the corrupted cookie.
In summary, when the app first launches, the background Titanium.Network.createHTTPClient call works, and any calls after that appear to send corrupted cookies.

The HTTPClient documentation says "object is intended to be used for a single request," so I assumed everything would reset after multiple calls. But something was different after the first call.
Adding loader.clearCookies(base_url); to the code before setting the cookies seems to fix the issue.
var loader = Titanium.Network.createHTTPClient();
loader.open("GET",base_url + url);
loader.onload = function() {
// process response
}
loader.clearCookies(base_url); // THE FIX.
loader.setRequestHeader('Cookie',Titanium.App.Properties.getString("cookies"));
loader.send();

WebView with Headers Connection Error

I am trying to open a Web view of a site (in Android) that needs token/Cookie in the headers for authentication.
I try to open the page using code below
HashMap<String,String> headers = new HashMap<String,String>();
headers.put("Cookie","MyToken");
MyWebView.loadUrl("https://myURL.com",headers);
I am hitting the onReceivedError with following values
errorCode: -6
*description: The connection to the server was unsuccessful.*
(PS: Since this site can be access from Intranet, using emulator I don't see this error the page loads correctly. Also HttpPost works fine with same token)

This is just a quick post about adding cookies to a web view. If you’ve ever tried to do this the way most people have said that it should be done, you’ve failed miserably and found this post. :)
The way it’s supposed to work is you set the cookie on the CookieManager and then tell the CookieSyncManager to sync.
CookieManager.getInstance().setCookie(domain, value);
CookieSyncManager.getInstance().sync();
I’ve never got this to work as described. With or without async tasks waiting for the threads to catch up.
Instead, I just add the cookie in the header of all the loadUrl calls.
Map<String, String> headers = new HashMap<String, String>();
headers.put("Cookie", "cookieName=cookieValue;domain=domain.com;path=/;Expires=Thu, 2 Aug 2021 20:47:11 UTC;");
webView.loadUrl("myurl.com", headers );
Caveat: I only need to initially load the appropriate cookie for the request, if you want to cover nested calls from inside the browser, you need to override shouldOverrideUrlLoading.
webView.setWebViewClient(new WebViewClient() {
#Override
public boolean shouldOverrideUrlLoading(WebView view, String url) {
view.loadUrl(url, headers);
return false;
}
});
If you need to inject the cookie for all requests(including images, js, etc), you’re going to need to override shouldInterceptRequest,

That is not a valid cookie header. Try:
headers.put("Cookie", "foo=bar");