Develop Reference

How to download a file protected by Cloudflare in Android WebView

I'm using Android WebView to download files from the URL entered by the user (including protected by Cloudflare), it looks something like this:
private final DownloadListener downloadListener = new DownloadListener() {
#Override
public void onDownloadStart(String url,
String userAgent,
String contentDisposition,
String mimeType,
long contentLength)
{
while (redirectionCount++ < MAX_REDIRECTS) {
HttpURLConnection conn = new URL(url).openConnection();
conn.setInstanceFollowRedirects(false);
conn.setConnectTimeout(...);
conn.setReadTimeout(...);
conn.setRequestHeader("User-Agent", WebSettings.getDefaultUserAgent(context));
conn.setRequestProperty("Accept-Encoding", "identity");
// Get the cookies for the current URL.
String cookiesString = CookieManager.getInstance().getCookie(url);
conn.setRequestProperty("Cookie", cookiesString);
...
}
}
};
I get the url from the WebView's DownloadListener, open the connection and pass the cookies obtained from WebView to the request header (cf_clearance=***; cf_chl_2=***; cf_chl_prog=***; cf_chl_rc_ni=***) and also WebView user-agent. JS and cookies are enabled in the WebView.
But the problem is that HttpURLConnection keeps returning HTTP 503 even if I pass the Cloudflare cookies, although the WebView seems to pass the Cloudflare checking. Maybe it requires something other than the cookies themselves?

Android Intercept a WebView request properly

My current code for intercepting a request in webview is
#Override
public WebResourceResponse shouldInterceptRequest(WebView view,
String url) {
String ext = MimeTypeMap.getFileExtensionFromUrl(url);
String mime = MimeTypeMap.getSingleton().getMimeTypeFromExtension(ext);
if (mime == null) {
return super.shouldInterceptRequest(view, url);
} else {
HttpURLConnection conn = (HttpURLConnection) new URL(
url).openConnection();
conn.setRequestProperty("User-Agent", userAgent);
return new WebResourceResponse(mime, "UTF-8",
conn.getInputStream());
}
}
I got this code from
The best way to intercept a WebView request in Android.
However, whenever I try to perform authentication, let's say I am loading facebook in my webview.
mWebView.loadUrl("https://www.facebook.com/");
Nothing is happening, what I noticed is that, the request headers are incomplete and also the response. Also, there are no cookies in the Sources. (I saw this when I remotely debugged the webview through Chrome).
Please correct me if I'm wrong, but I think that the incomplete headers and missing cookies is what causing the login request to fail.
Is there a way where I can modify the request and set its headers? Also for the response, should I do it too? And finally, how will I be able to have the cookies.

This question hasn't been answered for 6 months, so I don't know whether you will still need this, but maybe someone else has a similar question.
request headers are incomplete
When using HttpURLConnection you will be responsible to set any request headers, you might need, but it is as simple as setting the User-Agent, which you already did: conn.setRequestHeader(header, value) or if you want to add and not overwrite a header value: conn.addRequestHeader(header, value)
Alternatively, you could use okhttp, a HTTP client, which should add default values for headers, that are commonly expected.
there are no cookies in the Sources
When intercepting the request, you will also be in charge for handling cookies. You could store the cookie manually, by parsing the headers from the response e.g.
public WebResourceResponse shouldInterceptRequest(WebView view,
String url) {
// do your stuff
conn.connect(); // required to tell that the connection should be established
String cookie = getCookieFromConnection(conn);
// do more stuff and return WebResourceResponse
}
/**
* iterates all headers, and finds "cookie" headers
* (there could be more than one)
* #return cookie (concatenated value of all the found cookies)
* or null if no cookie has been found
*/
private String getCookieFromConnection(HttpURLConnection connection) {
String cookie = "";
Map<String, List<String>> header = connection.getHeaderFields();
List<String> cookies = header.get(COOKIE_HEADER);
if (cookies != null) {
for (String c : cookies) {
if (c != null) cookie += c + ";";
}
}
if (cookie.isEmpty()) return null;
return cookie;
}
or you could use a CookieManager, which would handle everything for you:
cookieManager = new CookieManager();
CookieHandler.setDefault(cookieManager);
cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
You could will also need to handle your cookies, when using okhttp, but again you could use the CookieManager as stated above. See this docu for more details, or this stackoverflow question.
Please correct me if I'm wrong, but I think that the incomplete headers and missing cookies is what causing the login request to fail.
There is another problem, when intercepting requests in a WebView: it somehow stops loading and evaluating javascript. I found this blog by Artem Zinnatullin online, who describes this behavior, and I experienced the same behavior.
If anyone would has a solution for this, I would be very happy.

Get the link redirected from webview android

I am trying to implement a payment gateway in android, and the payment processor requires sending some parameters when the "Pay" button is clicked. The sample link is:
https://vpay.com/?p=linkToken&v_merchant_id=qa331322179752&merchant_ref=234-567-890&memo=Bulk+order+from+McAckney+Web+Shop&total=13000&
notify_url=http%3A%2F%2Fwww.example.com%2Fnotification.php&
success_url=http%3A%2F%2Fwww.example.com%2Fthank_you.html&fail_url=http%3A%2F%2Fwww.example.com%2Ffailed.html
Now if the parameters are inputted correctly, the link returns another link in the format: https://vpay.com/pay/bnlink/xxxxxxxx-x0 which when visited brings up VPay payment page that can be used for payment based on the parameters supplied.
The payment processor should have employed an automatic redirection when the new link is generated, instead, it just displays the new link and stays there. Is there a way to get this new "RETURNED" link and then visit it so users can input the payment info.
Thank you!

Following method of Webview will work if you are using your own webview in activity or fragment for the payment stuff. below is the piece of code which will help:
webview.setWebViewClient(new WebViewClient() {
public boolean shouldOverrideUrlLoading(WebView view, String url) {
Log.i(TAG, "Processing webview url click...");
view.loadUrl(url);
Log.v(TAG,"html content of url:"+ getHtml(String url) );
return true;
}
public void onPageFinished(WebView view, String url) {
Log.i(TAG, "Finished loading URL: " +url);
}
});
in shouldOverrideUrlLoading method you will add your condition according to the url whether to load this url in webview or not.
And to get html content of the url use below Method:
public String getHtml(String url) {
HttpClient vClient = new DefaultHttpClient();
HttpGet vGet = new HttpGet(url);
String response = "";
try {
ResponseHandler<String> vHandler = new BasicResponseHandler();
response = vClient.execute(vGet, vHandler);
} catch (Exception e) {
e.printStackTrace();
}
return response;
}

Intercept POST requests in a WebView

I'm developping an Android application filtering the requests (with a white list) and using a custom SSLSocketFactory. For this, I've developed a custom WebViewClient and I have overridden the shouldInterceptRequest method. I can filter and use my SocketFactory with the GET requests but I can't intercept the POST requests.
So, is there a way to intercept the POST requests in a WebView ?
Here is the code of the shouldInterceptRequest method :
public final WebResourceResponse shouldInterceptRequest(WebView view, String urlStr) {
URI uri = URI.create(urlStr);
String scheme = uri.getScheme();
// If scheme not http(s), let the default webview manage it
if(!"http".equals(scheme) && !"https".equals(scheme)) {
return null;
}
URL url = uri.toURL();
if(doCancelRequest(url)) {
// Empty response
Log.d(TAG, "URL filtered: " + url);
return new WebResourceResponse("text/plain", "UTF-8", new EmptyInputStream());
} else {
Log.d(TAG, "URL: " + url);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("User-Agent", mSettings.getUserAgentString());
// Configure connections
configureConnection(conn);
String mimeType = conn.getContentType();
String encoding = conn.getContentEncoding();
if(mimeType != null && mimeType.contains(CONTENT_TYPE_SPLIT)) {
String[] split = mimeType.split(CONTENT_TYPE_SPLIT);
mimeType = split[0];
Matcher matcher = CONTENT_TYPE_PATTERN.matcher(split[1]);
if(matcher.find()) {
encoding = matcher.group(1);
}
}
InputStream is = conn.getInputStream();
return new WebResourceResponse(mimeType, encoding, is);
}
}

I was facing the same issue a few days ago.
So I built a library that solves it:
https://github.com/KonstantinSchubert/request_data_webviewclient
It is a WebViewClient with a custom WebResourceRequest that contains the POST/PUT/... payload of XMLHttpRequest requests.
It only works for these though - not for forms and other kind of request sources.
The hack works, basically, by injecting a script into the HTML that intercepts XMLHttpRequest calls. It records the post/put/... content and sends it to an android.webkit.JavascriptInterface. There, the request is stashed until the shouldInterceptRequest method is called by Android ...

I have created a library that aims to capture all data of all HTTP requests sent from Android WebViews.
Using this library, you can easily implement sending POST, or any other requests. Here's your code adapted to work with the library:
#Nullable
#Override
public final WebResourceResponse shouldInterceptRequest(WebView view, WebViewRequest request) {
URI uri = URI.create(request.getUrl());
String scheme = uri.getScheme();
// If scheme not http(s), let the default webview manage it
if(!"http".equals(scheme) && !"https".equals(scheme)) {
return null;
}
URL url = uri.toURL();
if(doCancelRequest(url)) {
// Empty response
Log.d(TAG, "URL filtered: " + url);
return new WebResourceResponse("text/plain", "UTF-8", new EmptyInputStream());
} else {
Log.d(TAG, "URL: " + url);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
// Set request method
conn.setRequestMethod(request.getMethod());
// Set request headers
for (Map.Entry<String, String> header : request.getHeaders().entrySet()) {
conn.setRequestProperty(header.getKey(), header.getValue());
}
// Set request body, if it's present
if (!request.getBody().isEmpty()) {
OutputStream os = conn.getOutputStream();
OutputStreamWriter osw = new OutputStreamWriter(os, "UTF-8");
osw.write(request.getBody());
osw.flush();
osw.close();
os.close(); //don't forget to close the OutputStream
}
conn.connect();
String mimeType = conn.getContentType();
String encoding = conn.getContentEncoding();
if(mimeType != null && mimeType.contains(CONTENT_TYPE_SPLIT)) {
String[] split = mimeType.split(CONTENT_TYPE_SPLIT);
mimeType = split[0];
Matcher matcher = CONTENT_TYPE_PATTERN.matcher(split[1]);
if(matcher.find()) {
encoding = matcher.group(1);
}
}
InputStream is = conn.getInputStream();
return new WebResourceResponse(mimeType, encoding, is);
}
}

I have one of my answers on above thread http://code.google.com/p/android/issues/detail?id=9122
Please see comment#31
Some of the caveats of my solution I see are:
Putting a dependency on xmlhttprequest prototype which has different implementation for different webkits.
Security issue in sending data for post requests in URL. But I guess you can solve that through some encryption mechanism.
URL length issue for some of the browsers if you big data to post
Apart from that, I found this github repo which seems to be solving this problem in another hacky way. I looked into the code but didn't get time to implement and test it. But worth giving a try.

The easiest way I found to do this is to just use the JQuery Ajax Event Handlers with a Javascript Interface.
You have to setup the Javascript Interface, but once you do just have it match the signature of the handler you're interested in.
To get the payload, just find the payload variable name put it in the interface.
Here's the Javascript:
$( document ).ajaxSend(function( event, request, settings ) {
var eventJSON= JSON.stringify(event);
var requestJSON = JSON.stringify(request);
var settingsJSON = JSON.stringify(settings);
var payloadJSON = JSON.stringify(payload);
myJSInterface.passData(eventJSON,requestJSON,settingsJSON,payloadJSON);
});
Here's the Kotlin class
class yourJSInterface() {
lateinit var event: String
lateinit var request: String
lateinit var settings: String
lateinit var payload: String
#JavascriptInterface
fun passData(eventJSON:String, responseJSON:String, settingsJSON:String,payloadJSON:String){
event = eventJSON
response= responseJSON
settings= settingsJSON
payload= payloadJSON
}
}
The registration in the onPageStarted override for WebViewClient
webView.addJavascriptInterface(yourJSInterface,"myJSInterface")
Finally the JS injection into the WebView in the OnPageFinished override
webView.evaluateJavascript("javascript:" + getString(R.string.js_from_above),null)
I registered the interface in onPageStarted because otherwise, the javascript file in onPageFinished won't recognize your interface.

There's a simpler solution: send the parameters by GET in the ajax call and transform them to POST in shouldInterceptRequest

you can get input value before submit
https://github.com/henrychuangtw/WebView-Javascript-Inject
Step 1 : create a class which called by javascript
class MyJavaScriptInterface
{
#JavascriptInterface
public void processHTML(String html)
{
//called by javascript
}
}
Step 2 : register interface for javascript
webview1.getSettings().setJavaScriptEnabled(true);
webview1.addJavascriptInterface(new MyJavaScriptInterface(), "MYOBJECT");
Step 3 : inject javascript to page
webview1.setWebViewClient(new WebViewClient() {
#Override
public void onPageFinished(WebView view, String url) {
super.onPageFinished(view, url);
StringBuilder sb = new StringBuilder();
sb.append("document.getElementsByTagName('form')[0].onsubmit = function () {");
sb.append("var objPWD, objAccount;var str = '';");
sb.append("var inputs = document.getElementsByTagName('input');");
sb.append("for (var i = 0; i < inputs.length; i++) {");
sb.append("if (inputs[i].type.toLowerCase() === 'password') {objPWD = inputs[i];}");
sb.append("else if (inputs[i].name.toLowerCase() === 'email') {objAccount = inputs[i];}");
sb.append("}");
sb.append("if (objAccount != null) {str += objAccount.value;}");
sb.append("if (objPWD != null) { str += ' , ' + objPWD.value;}");
sb.append("window.MYOBJECT.processHTML(str);");
sb.append("return true;");
sb.append("};");
view.loadUrl("javascript:" + sb.toString());
}
});

Use GET instead of POST.
Known issue:
http://code.google.com/p/android/issues/detail?id=9122
Was answered here as well:
Android - how to intercept a form POST in android WebViewClient on API level 4

Android - Webview only applying headers to initial request

I'm writing an android app that uses webview to request content from a web server, but using mWebView.loadUrl(url1, headers); will only apply the headers to the initial request and not the resources in the request.
Any idea as so how to apply the headers to the resource requests as well?

Not absolutely sure but you can try to override shouldOverrideUrlLoading(WebView view, String url) method and handle all redirects by starting mWebView.loadUrl(url, yourHeaders);
Dont forget to return true in that overriden method.

First of all, let me say that i can't believe that webview sucks so much.
This is what i did to pass custom headers
public class CustomWebview extends WebView {
public void loadWithHeaders(String url) {
setWebViewClient(new WebViewClient() {
#Override
public WebResourceResponse shouldInterceptRequest(WebView view, String url) {
//makes a custom http request, which allows you to add your own headers
return customRequest(url);
}
});
loadUrl(url);
}
/**
* Custom http request with headers
* #param url
* #return
*/
private WebResourceResponse customRequest(String url) {
try {
OkHttpClient httpClient = new OkHttpClient();
Request request = new Request.Builder()
.url(url.trim())
.addHeader("Header-Name", "Android Sucks")
.build();
Response response = httpClient.newCall(request).execute();
return new WebResourceResponse(
"text/html", // You can set something other as default content-type
"utf-8", // Again, you can set another encoding as default
response.body().byteStream()
);
} catch (IOException e) {
//return null to tell WebView we failed to fetch it WebView should try again.
return null;
}
}
}