Develop Reference

How to crawl cellphone apps that use HTTP connect method with Fiddler?

I'm new to web-crawlers, trying to crawl ridership data of metro from the cellphone maps app(www.amap.com) with Fiddler, but I got this HTTP connect method, which is not viewable. There are icons of locks next to the URL and in 'Response' it says this:
'Encrypted HTTPS traffic flows through this CONNECT tunnel. HTTPS Decryption is enabled in Fiddler, so decrypted sessions running in this tunnel will be shown in the Web Sessions list.'
I found a solution suggesting that customizing rules in fiddler may help, so I followed and added this to its script:
if (oSession.oRequest[‘User- Agent’].IndexOf("Android") > -1 && oSession.HTTPMethodIs("connect")) {
oSession.oResponse.headers["Connection"]="Keep-Alive";
}
The changes to Fiddler Script
But of course, it didn't work, I've tried both iPhone and android and changed the header in the script respectively, none of them helped.
So is this app and HTTP connect method crawlable? The data is constructively helpful to my research, instead, it is not provided in website 'amap', so it has to be done through a cellphone.

If you have HTTPS decryption enabled in Fiddler but you see (mostly) only CONNECT requests this means that the apps on the device try to open a connection but do not trust the Fiddler root certificate.
If you try to use the apps on-device you will notice that there is currently no working network connection available (requests just don't work as the apps don't accept the server certificate created by Fiddler).
On Android devices since Andorid 6 you need root permissions to instal the Fiddler rot certificate or alternatively if you want to monitor a single app you can try to modify and re-sign the app. All details are described in this question and answer:
Some androids apps won't connect through fiddler

Ionic $http request get not working with error data null status -1

I use Ionic and Cordova framework for developing a mobile app, I've got the issue that can not use http get to request some url. For $http.get to some url that works fine, but some others not work and it response error like as
I/Web Console( 9179): {"data":null,"status":-1,"config":{"method":"GET","transformRequest":[null],"transformResponse":[n ull],"url":"http://someexmaple.com/","headers":{"Accept":"application/json, text/plain, */*"},"withCredentials":true}," statusText":""}
This issue only happens when I test on real mobile device and using adb debugging tool. But I tested on Browser and Genymotion virtual devices, both of them worked well for all url request.
So I've know idea with this response error. I think may be my device got problem.

Finally I've resolved my problem. I use inspect tool in Chrome, and move to network tab for watching the network packet, I see when I access to the url of site, it redirect me to another server port (instead of 80, it redirect to 443), I've noticed this because the 'Remote Address field' is the same address of url server, but port was changed. So I changed my $http.get request url from port 80 into port 443. And then it works, so I guess that when $http.get to the url but it didn't redirect to another port to get data, so it just response the null.

how to access webpage hosted on tomcat server from the android mobile

I have designed a webpage and run that on my browser using tomcat server http://localhost:8084/neclogin/main.jsp
I also accessed it from my android emulator it works well.
http://10.0.2.2:8084/neclogin/main.jsp it works well.
But when I tried on my real device (connected to PC via hotspot)
http://192.xxx.xxx.xxx:8084/neclogin/web/main.jsp
It shows on browser that HTTTP 404 ERROR - DESCRIPTION :requested resource is unavailable
I am confused whether my android mobile is unable to connect to localhost or can't trace the path of my main.jsp
FOR MORE DETAIL
1)windows 7
2)NETBEANS IDE WITH APACHE TOMCAT SERVER 8.03
3)**Both connected via wifi hotspot on my mobile**
4)path to my main.jsp is `C:\Users\dell\Documents\NetBeansProjects\neclogin\web\main.jsp`
5)path to my tomcat server installed is `C:\Program Files\Apache Software Foundation\Apache Tomcat 8.0.3`
i have disabled my windows and antivirus firewall

First try to access http://192.xxx.xxx.xxx:8084/neclogin/web/main.jsp from your PC to see if the IP is correct.
a) If that works and you can access it, then indeed there is a problem with the external access and the problem is somewhere in your firewall settings or alike. Potentially it doesn't let traffic through 'non-standard' ports (80,433 etc)
b) If you can't access it on your PC either using then you got the IP wrong. Try running ipconfig from the command line and see if with the IP listed there it works. ipconfig will return a bunch of different IPs you need to look for the one that starts with 192 most likely labeled as IPv4 or something like that.
It's also worth checking what the 404 message looks like. If it's tomcat's 404 page then you're almost there as you can access tomcat but not using the correct path.

Access to asp.net web service

I wrote a little WS on asp.net, I can open it printing something like
http://46.146.170.225/RouteGen/Service.asmx
in address bar. It's all right, WS works.
But if I print the same address in a browser on the other computer, the page isn't available. How to get access to my web server from other PC? (I need from Android device, but I think it's no difference)

If you started the Web-Service from within Visual Studio then without changing the starup-settings of your project - it's not possible, because VS only starts a local debug web-server that doesn't allow calls from other hosts than localhost.
To allow external IPs to access your web-server, you have to set up the IIS and run your web-service inside it. A firewall could block incoming requests to the IIS but I ran such a service last winter and didn't have to change firewall-settings.

Verify that the website, in IIS, is bound to a public-facing IP address. Right click on your website in IIS, and go to the bindings setting. Then, check the host field. It should have an IP address or domain name that is available publicly.
Verify that your firewall has Port 80 open for incoming traffic

How to debug http calls on Android devices?

I'm writing a Lovefilm client for Android, and it's not going too badly except I keep having problems with the remote calls to retrieve data from the API.
Does anyone have any tips for debugging remote calls like this? Can I tcpdump on Android or is there a native way of doing it?
For example, I'm using the Scribe-java library for OAuth to access the Lovefilm API, I can authenticate find and retrieve a list of films on the users account fine when the device is running Gingerbread, but trying to retrieve the accessToken on Froyo causes a blank response & and apparent response code of -1, I'd like to be able to see what's going on under the cvers their.
Another example I'd like to be able to the raw http for is trying to run a search, I get and IOError that says "Received authentication challenge is null"

I've used Fiddler (http-proxy for debugging http calls) with the android emulator in these cases. Just start the proxy, and start the emulator with the correct proxy address (-http-proxy ).

Fiddler is the most useful option. On the emulator #Scythe answer will work, but on a real device you will need to set the proxy in the Apache Http Client. The following code will do that:
HttpHost proxy = new HttpHost("youripaddr", 8888);
params.setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
If you are using https, fiddler is not so useful. In that case can enable the build in logging support in Apache Http Client. The following code does that:
Headers only:
java.util.logging.Logger apacheHeaderLog = java.util.logging.Logger.getLogger("org.apache.http.headers");
apacheHeaderLog.setLevel(java.util.logging.Level.FINEST);
Headers & Wire:
java.util.logging.Logger apacheWireLog = java.util.logging.Logger.getLogger("org.apache.http.wire");
apacheWireLog.setLevel(java.util.logging.Level.FINEST);
Note that this will have to have a java.util.logging Handler configured at finest level and the default handler is configured to log to logcat, which will filter DEBUG (finest) entries by default.

If your system can share the wi-fi connection you should be able to route packets from any device through your system and then using wireshark you can get monitor your calls or get a tcpdump.
Also , and more importantly , it would be best if you log your network calls and responses as suggested by #Matthew
Windows 7 wi-fi connection sharing : http://www.winsupersite.com/article/faqtip/windows-7-tip-of-the-week-use-wireless-hosted-networking-to-share-an-internet-connection-wirelessly.aspx

Since I always run into similar troubles and it seems a lot of people having the same issues over and over again I wrote up a quick tutorial for debugging client-server communication by using netcat and cURL.
That of course only works for the simplified case that you always 'fake' on side of the connection.
For eavesdropping you can use tools like tcpdump or Wireshark. Which will definitely be easier if you're able to run the server instance directly on your local machine.

Stetho is a great tool from FB which helps in debugging android Apps. You can have access to local data and have a check on your network using this.
http://facebook.github.io/stetho/