Develop Reference

How to crawl cellphone apps that use HTTP connect method with Fiddler?

I'm new to web-crawlers, trying to crawl ridership data of metro from the cellphone maps app(www.amap.com) with Fiddler, but I got this HTTP connect method, which is not viewable. There are icons of locks next to the URL and in 'Response' it says this:
'Encrypted HTTPS traffic flows through this CONNECT tunnel. HTTPS Decryption is enabled in Fiddler, so decrypted sessions running in this tunnel will be shown in the Web Sessions list.'
I found a solution suggesting that customizing rules in fiddler may help, so I followed and added this to its script:
if (oSession.oRequest[‘User- Agent’].IndexOf("Android") > -1 && oSession.HTTPMethodIs("connect")) {
oSession.oResponse.headers["Connection"]="Keep-Alive";
}
The changes to Fiddler Script
But of course, it didn't work, I've tried both iPhone and android and changed the header in the script respectively, none of them helped.
So is this app and HTTP connect method crawlable? The data is constructively helpful to my research, instead, it is not provided in website 'amap', so it has to be done through a cellphone.

If you have HTTPS decryption enabled in Fiddler but you see (mostly) only CONNECT requests this means that the apps on the device try to open a connection but do not trust the Fiddler root certificate.
If you try to use the apps on-device you will notice that there is currently no working network connection available (requests just don't work as the apps don't accept the server certificate created by Fiddler).
On Android devices since Andorid 6 you need root permissions to instal the Fiddler rot certificate or alternatively if you want to monitor a single app you can try to modify and re-sign the app. All details are described in this question and answer:
Some androids apps won't connect through fiddler

Cordova - Android HTTPS requests fail on 4G

I've created Cordova Android app, and I'm facing this issue:
When on 3G/4G, whenever I try to make request towards my server I get the error:
ERR_TUNNEL_CONNECTION_FAILED.
When I am on WIFI everything works fine. This is not happening on all Android devices, I am facing this issue on Samsung Galaxy A5 (nd some other Androids).
I've discovered that if we use HTTP instead of HTTPS everything is fine.
Also according to https://www.sslshopper.com/ssl-checker.html
and
https://www.digicert.com/help/
everything seems to be fine with our SSL cert.
How can we solve this issue?

This usually happens because your provider is configuring a proxy in your device.
Try checking your current APN setting and deleting the fields proxy and port from it.
This also can be caused if you are using a port different from 443

I wonder if you have found your answer or not, but still want to post answer for others looking for the solution: Note that Cordova doesn't allow https calls to Servers with untrusted ssl certificate installed on them. You can ignore this error and continue by making a small change in a cordova file.
Open “\cordova\platforms\android\CordovaLib\src\org\apache\cordova\
CordovaWebViewClient.java”. In 'onReceivedSslError' method, comment
the else part and add handler.proceed() instead.

This issues happens with mobile data connection because the default Access Point that came with carrier generally have proxy that will not allowing SSL Tunneling. All you have to do is to set Proxy and Port to nothing.
Also ensure to use only port 443 for HTTPS.

HTTPS url not encrypted

I'm using Charles proxy to fetch all the requests coming from my Android app to a webservice.
The thing is Charles shows me the complete request, meaning I can see the whole URL, headers and body so I can see www.example.com/rest/resource/param1/param2,
the JSON I send with it and also the authentication header.
After reading several posts like this and this one I thought the good part of working with the TLS was that one could only get the domain name from the URL, in this case www.example.com
To make sure it's not the client's fault, I requested the webservice resource with Retrofit and HttpsURLConnection and I could see the whole request both times.
I guess also the certificate is properly installed because it is shown in the browser every time an https request is made. Am I missing something else here or is this the normal behaviour?
So far I couldn't find a reason for this to happen so any help will be appreciated.

To debug with Charles proxy you must install a certificate on your browser (client).
With https the URL is encrypted.
But because you choose to use that proxy, your browser establish a secure connection to that proxy, and the proxy to the website. So, only 1) you, 2) the proxy 3)the website can decrypt the https traffic.
By installing a CA certificate on your browser, you allow the person detaining the corresponding private key (in your case, your proxy) to impersonate (so, decrypt with a MITM) any website.

Why does android get the wrong ssl certificate? (two domains, one server)

I have two domains: foo.net and bar.com. They both have SSL certificates, and they work well in all desktop and mobile browsers. They are hosted on the same server configured with nginx.
However, when I make a request to a domain from within a native android app, it somehow gets the certificate from the wrong domain! This results in an IO Exception:
request = new HttpPost("https://foo.net/api/v1/baz");
request.setHeader("Authorization", "user:pass");
response = httpClient.execute(request);
...
javax.net.ssl.SSLException: hostname in certificate didn't match: <foo.net> != <bar.com> OR <bar.com> OR <www.bar.com>
What would cause android/java to try using the certificate from bar.com when every other measure seems to indicate that the server is correctly configured? Nothing appears in the nginx access or error log. There is no mention of bar.com anywhere in my android project.
Edit: I'm not sure why, but it appears that the server is using the certificate for bar.com for the server IP https://198.245.xx.xxx

The most likely cause for this problem is that the server uses Server Name Indication to choose which certificate to send. If the client doesn't support SNI, the server cannot choose which certificate to send during the SSL/TLS handshake (before any HTTP traffic is sent). SNI is required when you want to use multiple certificates on the same IP address and port, but not all clients support it (notoriously, IE on any version of Windows XP, and a number of mobile browsers).
You're also visibly using the Apache HTTP Client library (not HttpsURLConnection, for which there can be SNI support with some Android versions.
Support for SNI in the Apache HTTP Client library is quite recent, and certainly hasn't made it into the Android stack.
You may find the workaround described in this article useful (although it seems only to work for Android 4.2+).
Another two options would be:
to use a distinct IP address for each host (so as not to need SNI), if you're in control of server, or
to use another HTTP Client library (e.g. HttpsURLConnection).

A solution for Apache, more like a trick:
the SSL certificates are loaded based on the vhost name from /etc/apache2/sites-enabled. So, to trick that check make sure the problematic certificate is loaded first (remember that the vhosts are loaded by name).

It looks like the certificate of foo.net is misconfigured, and is using the same hostname as bar.com
Try to run an online certificate validation tool, like https://www.digicert.com/help/ on foo.net, just to be sure.
I think that you need to regenerate the certificate of foo.net with the right hostname, or reconfigure ngix to make sure that nginx serve the right certificate for the right host.

How to debug http calls on Android devices?

I'm writing a Lovefilm client for Android, and it's not going too badly except I keep having problems with the remote calls to retrieve data from the API.
Does anyone have any tips for debugging remote calls like this? Can I tcpdump on Android or is there a native way of doing it?
For example, I'm using the Scribe-java library for OAuth to access the Lovefilm API, I can authenticate find and retrieve a list of films on the users account fine when the device is running Gingerbread, but trying to retrieve the accessToken on Froyo causes a blank response & and apparent response code of -1, I'd like to be able to see what's going on under the cvers their.
Another example I'd like to be able to the raw http for is trying to run a search, I get and IOError that says "Received authentication challenge is null"

I've used Fiddler (http-proxy for debugging http calls) with the android emulator in these cases. Just start the proxy, and start the emulator with the correct proxy address (-http-proxy ).

Fiddler is the most useful option. On the emulator #Scythe answer will work, but on a real device you will need to set the proxy in the Apache Http Client. The following code will do that:
HttpHost proxy = new HttpHost("youripaddr", 8888);
params.setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
If you are using https, fiddler is not so useful. In that case can enable the build in logging support in Apache Http Client. The following code does that:
Headers only:
java.util.logging.Logger apacheHeaderLog = java.util.logging.Logger.getLogger("org.apache.http.headers");
apacheHeaderLog.setLevel(java.util.logging.Level.FINEST);
Headers & Wire:
java.util.logging.Logger apacheWireLog = java.util.logging.Logger.getLogger("org.apache.http.wire");
apacheWireLog.setLevel(java.util.logging.Level.FINEST);
Note that this will have to have a java.util.logging Handler configured at finest level and the default handler is configured to log to logcat, which will filter DEBUG (finest) entries by default.

If your system can share the wi-fi connection you should be able to route packets from any device through your system and then using wireshark you can get monitor your calls or get a tcpdump.
Also , and more importantly , it would be best if you log your network calls and responses as suggested by #Matthew
Windows 7 wi-fi connection sharing : http://www.winsupersite.com/article/faqtip/windows-7-tip-of-the-week-use-wireless-hosted-networking-to-share-an-internet-connection-wirelessly.aspx

Since I always run into similar troubles and it seems a lot of people having the same issues over and over again I wrote up a quick tutorial for debugging client-server communication by using netcat and cURL.
That of course only works for the simplified case that you always 'fake' on side of the connection.
For eavesdropping you can use tools like tcpdump or Wireshark. Which will definitely be easier if you're able to run the server instance directly on your local machine.

Stetho is a great tool from FB which helps in debugging android Apps. You can have access to local data and have a check on your network using this.
http://facebook.github.io/stetho/