I'm new to web-crawlers, trying to crawl ridership data of metro from the cellphone maps app(www.amap.com) with Fiddler, but I got this HTTP connect method, which is not viewable. There are icons of locks next to the URL and in 'Response' it says this:
'Encrypted HTTPS traffic flows through this CONNECT tunnel. HTTPS Decryption is enabled in Fiddler, so decrypted sessions running in this tunnel will be shown in the Web Sessions list.'
I found a solution suggesting that customizing rules in fiddler may help, so I followed and added this to its script:
if (oSession.oRequest[‘User- Agent’].IndexOf("Android") > -1 && oSession.HTTPMethodIs("connect")) {
oSession.oResponse.headers["Connection"]="Keep-Alive";
}
The changes to Fiddler Script
But of course, it didn't work, I've tried both iPhone and android and changed the header in the script respectively, none of them helped.
So is this app and HTTP connect method crawlable? The data is constructively helpful to my research, instead, it is not provided in website 'amap', so it has to be done through a cellphone.
If you have HTTPS decryption enabled in Fiddler but you see (mostly) only CONNECT requests this means that the apps on the device try to open a connection but do not trust the Fiddler root certificate.
If you try to use the apps on-device you will notice that there is currently no working network connection available (requests just don't work as the apps don't accept the server certificate created by Fiddler).
On Android devices since Andorid 6 you need root permissions to instal the Fiddler rot certificate or alternatively if you want to monitor a single app you can try to modify and re-sign the app. All details are described in this question and answer:
Some androids apps won't connect through fiddler
Related
I am working on the Android client implementation that connects via websocket to a server. I've just enabled HTTPS on the server instance and now I am receiving this issue:
W/System.err: com.neovisionaries.ws.client.HostnameUnverifiedException: The certificate of the peer (CN=otherdomain.com) does not match the expected hostname (domain.com)
at com.neovisionaries.ws.client.SocketConnector.verifyHostname(SocketConnector.java:171)
at com.neovisionaries.ws.client.SocketConnector.doConnect(SocketConnector.java:126)
at com.neovisionaries.ws.client.SocketConnector.connect(SocketConnector.java:83)
at com.neovisionaries.ws.client.WebSocket.connect(WebSocket.java:2152)
at com.neovisionaries.ws.client.ConnectThread.runMain(ConnectThread.java:32)
at com.neovisionaries.ws.client.WebSocketThread.run(WebSocketThread.java:45)
It seems that the websocket client is finding mismatched server host names.
Some other related info:
otherdomain.com is a different one of our URLs. It is pointed to separate instance of the same web application running on the same server as domain.com which is the app I am currently working on.
My code is logging the URL that it is connecting to and I can see that it is correctly referencing domain.com:
connecting to: wss://domain.com?session_key=TheCorrectSessionKey
As noted otherdomain.com is another instance of the same web application. That instance already previously had HTTPS enabled and everything is working properly with that instance. The same client code I am using now is able to connect successfully to this instance.
The same web application has a front end page that also connects to the websocket via wss://. This is working properly in both Chrome and Firefox browsers for domain.com, as far as I can tell the only client having issues is the Android client, and it's only the domain.com variant of the app having issues, the otherdomain.com one seems to be working fine.
I found this issue opened on the library I am using for websockets. But I'm not 100% sure if my problem is related to it, but it may be. In particular someone states:
Not being able to connect to a WebSocket-enabled server domain unless it is also configured to be the default server on the website.
I assume only one site can be default, and perhaps otherdomain.com got set to default since it was completed first. This is all a bit over my head though to be honest, I'm not sure if this is relavent or not.
I've created Cordova Android app, and I'm facing this issue:
When on 3G/4G, whenever I try to make request towards my server I get the error:
ERR_TUNNEL_CONNECTION_FAILED.
When I am on WIFI everything works fine. This is not happening on all Android devices, I am facing this issue on Samsung Galaxy A5 (nd some other Androids).
I've discovered that if we use HTTP instead of HTTPS everything is fine.
Also according to https://www.sslshopper.com/ssl-checker.html
and
https://www.digicert.com/help/
everything seems to be fine with our SSL cert.
How can we solve this issue?
This usually happens because your provider is configuring a proxy in your device.
Try checking your current APN setting and deleting the fields proxy and port from it.
This also can be caused if you are using a port different from 443
I wonder if you have found your answer or not, but still want to post answer for others looking for the solution: Note that Cordova doesn't allow https calls to Servers with untrusted ssl certificate installed on them. You can ignore this error and continue by making a small change in a cordova file.
Open “\cordova\platforms\android\CordovaLib\src\org\apache\cordova\
CordovaWebViewClient.java”. In 'onReceivedSslError' method, comment
the else part and add handler.proceed() instead.
This issues happens with mobile data connection because the default Access Point that came with carrier generally have proxy that will not allowing SSL Tunneling. All you have to do is to set Proxy and Port to nothing.
Also ensure to use only port 443 for HTTPS.
Scenario: I am trying to debug an Android app by proxying requests through Fiddler.
I got FiddlerRoot certificate installed on the Android device, and the SSL decryption works for most requests, but for other requests I can only see the HTTPS Connect, and nothing else in the Fiddler log. I think it might be image requests over SSL that fails to decrypt.
I have double-checked that "Hide images" is off, etc. Images retrieved are hosted on another domain than the main API the app talks to.
What could cause this behaviour ? And how do I get the image requests to show in Fiddler ?
I am using the latest Fiddler4.
There are plenty of tutorials on how you can intercept HTTP(s) traffic from Android using Fiddler.
Try this one: http://docs.telerik.com/fiddler/configure-fiddler/tasks/configureforandroid
However, it will fail when you try to intercept and decrypt Android SSL traffic coming from an application, and not from a browser.
It might be that the application uses a certificate pinning – and you are probably cannot decipher this connection. Lost cause!
But more probably, the reason is a bug in the HttpsUrlConnection pipeline implementation.
To solve the issue, please proceed with the following steps:
In Fiddler click "Rules->Customize Rules";
Find function OnBeforeResponse in the script
Add following code to the function body:
if (oSession.oRequest["User-Agent"].indexOf("Dalvik") > -1 &&
oSession.HTTPMethodIs("CONNECT")) {
oSession.oResponse.headers["Connection"] = "Keep-Alive";
}
Save the file and restart Fiddler.
I am creating an aplication that involves an WebView. The thing is that I want to load the full page and not the mobile one, so I have changed the User Agent. Nevertheless there are pages that loads the mobile version.
Here are two versions of code that I have tried:
1.webview.getSettings().setUserAgentString("Mozila ");
2.
String DESKTOP_USERAGENT = webview.getSettings().getUserAgentString ();
DESKTOP_USERAGENT = DESKTOP_USERAGENT.replace("Mobile ","");
webview.getSettings().setUserAgentString(DESKTOP_USERAGENT);
This are exemples of webpages that loads the mobile version in any cases:
http://www.jurnalul.ro
http://www.androidzoom.com
1.Does anyone knows how I can trick the server and tell him I am using a desktop and not a mobile?
2.How does a website knows that I am using a mobile version?
Thank you very much,
Razvan
The problem may be that if you are using a device that your carrier is routing all your HTTP requests through a proxy, and that the proxy is changing the User-Agent. Check on the other end, with your own server, using nc -l 80 -vvv that your request is indeed sending the User-Agent that you have modified.
EDIT: Some specific troubleshooting steps
Forward a port 9090 on your router to your desktop machine or laptop.
Download netcat
Run netcat with the command "nc -l 9090 -vvv"
In your Android application's WebView, make an HTTP request with the User-Agent you are injecting to http://your.ip.address:9090
In the terminal you ran netcat, you will see the HTTP request dump in plain text. There you can check the HTTP header User-Agent to see if it has been changed by a proxy server or not.
You cannot test this stuff with Wireshark or Fiddler because it is happening in the WAN. You need to test it on the receiving end, either on a server, or on your own desktop machine.
webview.getSettings().setUserAgent(1);//for desktop 1 or mobil 0.
I'm writing a Lovefilm client for Android, and it's not going too badly except I keep having problems with the remote calls to retrieve data from the API.
Does anyone have any tips for debugging remote calls like this? Can I tcpdump on Android or is there a native way of doing it?
For example, I'm using the Scribe-java library for OAuth to access the Lovefilm API, I can authenticate find and retrieve a list of films on the users account fine when the device is running Gingerbread, but trying to retrieve the accessToken on Froyo causes a blank response & and apparent response code of -1, I'd like to be able to see what's going on under the cvers their.
Another example I'd like to be able to the raw http for is trying to run a search, I get and IOError that says "Received authentication challenge is null"
I've used Fiddler (http-proxy for debugging http calls) with the android emulator in these cases. Just start the proxy, and start the emulator with the correct proxy address (-http-proxy ).
Fiddler is the most useful option. On the emulator #Scythe answer will work, but on a real device you will need to set the proxy in the Apache Http Client. The following code will do that:
HttpHost proxy = new HttpHost("youripaddr", 8888);
params.setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
If you are using https, fiddler is not so useful. In that case can enable the build in logging support in Apache Http Client. The following code does that:
Headers only:
java.util.logging.Logger apacheHeaderLog = java.util.logging.Logger.getLogger("org.apache.http.headers");
apacheHeaderLog.setLevel(java.util.logging.Level.FINEST);
Headers & Wire:
java.util.logging.Logger apacheWireLog = java.util.logging.Logger.getLogger("org.apache.http.wire");
apacheWireLog.setLevel(java.util.logging.Level.FINEST);
Note that this will have to have a java.util.logging Handler configured at finest level and the default handler is configured to log to logcat, which will filter DEBUG (finest) entries by default.
If your system can share the wi-fi connection you should be able to route packets from any device through your system and then using wireshark you can get monitor your calls or get a tcpdump.
Also , and more importantly , it would be best if you log your network calls and responses as suggested by #Matthew
Windows 7 wi-fi connection sharing : http://www.winsupersite.com/article/faqtip/windows-7-tip-of-the-week-use-wireless-hosted-networking-to-share-an-internet-connection-wirelessly.aspx
Since I always run into similar troubles and it seems a lot of people having the same issues over and over again I wrote up a quick tutorial for debugging client-server communication by using netcat and cURL.
That of course only works for the simplified case that you always 'fake' on side of the connection.
For eavesdropping you can use tools like tcpdump or Wireshark. Which will definitely be easier if you're able to run the server instance directly on your local machine.
Stetho is a great tool from FB which helps in debugging android Apps. You can have access to local data and have a check on your network using this.
http://facebook.github.io/stetho/