I have installed and have been using charles proxy(4.5.6).
I am using Android 9.
Charles works on the Chrome pages (Instagram specifically):
but it does not work on the Instagram application:
and I always get that error when I am using the app.
Should I "Save Charles root certificate" and then install it on my phone? Also, why does Charles work on the Chrome but not on the Instagram app?
Instagram's app uses a technique known as SSL Certificate Pinning. Basically, this means they bundle information about the expected SSL certificate for the server into the app, and deny any connection that doesn't present that certificate. Because the certificate presented by Charles is not Instagram's certificate, the connection is denied by the Instagram app. In Chrome, it must rely on the browser's assessment of certificate trust, which will defer to certificates installed on the device - I assume you've already set up your device to trust the Charles Proxy SSL certificate, so because Chrome trusts it (because you told it to), the website works. There used to be a way to do key pinning in the browser, but it no longer works due to some issues that were discovered.
You can disable this certificate pinning in your Facebook account settings: https://www.facebook.com/whitehat/researcher-settings/
As for the details of their implementation, there are numerous articles online about how to disable the pinning in Instagram's app. I can't vouch for any of them in particular, as I haven't tried them, and I'm not sure how they would interact with the terms of service, but you could take a look at those for more information.
I have been working with Charles to monitor and debug network requests for my app and I would like to mention few points that would help you:
If you look at the documentation of Charles SSL Certificate for Android,
As of Android N, you need to add configuration to your app in order to have it trust the SSL certificates generated by Charles SSL Proxying. This means that you can only use SSL Proxying with apps that you control.
This restriction has been added by the Android framework itself to avoid exploits and hacks for more security. So, you cannot use Charles to montior or debug network requests of third party apps (in your case -Instagram) if you're having Android 7 (Nougat) or higher.
If you really want to do it, you can try connecting an Android phone with Android 6 (Marshmallow) or lower to see if it works for you.
If you want to debug your own app, your phone and the system running Charles should be on the same network and you need to setup proxy configuration in your phone's Wi-Fi settings.
First go to Charles -> Help -> SSL Proxying -> Install Charles Root Certificate on Mobile Device or Remote Browser
You'll see an info window like this:
Note the IP address mentioned in this window and go to your phone's Settings -> Wi-Fi -> Select your Wi-Fi network -> Edit
You'll find option called Proxy, it will be None by default, update it to Manual. You'll get two input fields for entering Hostname and Port, fill those details by looking at the IP mentioned in the Charles and Save it.
Another important point is, you'll need to install Charles Root Certificate on your phone. You can do it by visiting https://chls.pro/ssl from your phone's browser. It'll automatically download the certificate and will prompt you to install it. Make sure you're connected to Charles by following the 2nd step, otherwise it won't download the certificate automatically.
Once it's done, you're good to go!
There may be two reason for this
1) The instagram app is using network security configuration file which disabled the proxy servers
for more information read this website https://developer.android.com/training/articles/security-config
2) For checking payload of your own application you need to set proxy server
as https://community.tealiumiq.com/t5/Tealium-for-Android/Setting-up-Charles-to-Proxy-your-Android-Device/ta-p/5121
Related
I have a Squid proxy installed with HTTPS/SSL. When I install the generated certificate in Firefox and set the proxy, everything looks good -- that is, I can see all the HTTPS requests the Squid log files.
However, when I install the same certificate in my Android phone (and change the proxy setting), it seems to work only in the browser. I first change the proxy setting and pages didn't load, then I installed the certificate and all was good.
All other app I've tried yield in errors due to connection problems. I can see the CONNECT request in the Squid logs but no other requests (e.g., GET, POST). I know that some app completely ignore the system proxy settings, but many use them.
The odd thing is that everything was working on an old Android 6.0 phone. Here the same apps where working perfectly fine using the proxy and relying on HTTPS requests. Now I have a newer Android 9.0 phone and all apps so far fail. The apps obviously use the proxy (seeing the CONNECT entries in the logs) but do not use/acknowledge the installed certificate.
I currently see the new phone or the new Android version as cause for the problem since it worked before with an older phone and Android version. How can I best track this issue down?
"Since Android 7, apps ignore user provided certificates, unless they are configured to use them. As most applications do not explicitly opt in to use user certificates [...] we need to place our CA certificate in the system certificate store"
https://docs.mitmproxy.org/stable/howto-install-system-trusted-ca-android/
However, this seems not to be easily done...
Android connects to specific domains to verify the connection.
Add this bit to the the squid.conf:
# Mobile
acl google-servers dstdom_regex "/etc/squid/google.txt"
always_direct allow google-servers
next, create this file /etc/squid/google.txt with the following content:
(^|\.)android\.clients\.google\.com$
(^|\.)ggpht\.com$
(^|\.)google\.com$
(^|\.)www\.googleapis\.com$
(^|\.)gstatic\.com$
(^|\.)gvt1\.com$
(^|\.)1e100\.net$
(^|\.)google\.co\.uk$
to activate, run:
./squid -k reconfigure
the older pre-3.10 versions of Charles allow users to install a root certificate on their phones to help apps to allow SSL connections but the newer version has removed this feature. I am not sure if this is the reason why I wasn't able to POST successfully to the native app server.
For example I will get messages like SSLHandshake: Remote host closed connection during handshake
I suspect the root certificate is for the app on my phone to accept SSL connections from the server, but not for me to POST messages to the server.
Anyways, is there a method for me to set up SSL connections to POST?
P.S. I have added the server's url in my SSL list, and also enabled "transparent HTTP proxy." (I noticed that is not HTTPS, so perhaps Charles doesn't have transparent HTTPS feature?)
Update: I tried using mitmproxy and it worked. It looks like installing cer file to the phone is the right way to go but I am wondering why Charles removed this feature. I also think I might have missed something in the documentation. Perhaps Charles did generate a cer file in my system for me to download to the phone. If so, where can I find this file?
Yeah, all you need to do is to click help on the menu bar and then select save SSL or install SSL on mobile device and then browse to the url it gives you on your mobile browser to download the certificate.
I am on a Windows PC and would like to monitor HTTPs traffic from my Android phone from a particular app. I installed Honeyproxy and set a proxy address on my android device using Proxy Settings app. I am able to see the GET traffic of my phone. Next, to get the HTTPS traffic, I installed the user trusted certificates as described here: http://wiki.cacert.org/FAQ/ImportRootCert#Android_Phones_.26_Tablets
I can confirm that I can see both certificates (root and class3) in the "user trusted certificates" list in Settings> Security. Next I ran the honeyproxy expecting to see the HTTPS traffic this time but there is none.
Further, no HTTPs link from the device opens (I get a certificate untrusted). What is the step I am missing? Do let me know your thoughts!
I have enabled HTTPS in Fiddler4 options, and it does can capture HTTPS communications from most Android Apps on my machine(With Android simulator, via WIFI proxy settings).
But for some Apps it always failed. e.g. Kayak.
It always says "Oops! There was a problem connecting to the internet. Please try again later.".
I notice Kayak App uses TLS 1.0(See following screenshot, it's from Microsoft Network Monitor 3.4), i think maybe this has something to do with it.
I also tried to set the protocols into "tls1.0"(See following screenshot), but has no effect.
Appreciate your ideas.
Update Further investigation revealed that some Android applications will not accept wildcards inside certificates' SubjectCN field if that field is encoded as BMPString. The makecert generator uses BMPString, so you can either untick the Use wildcards box or switch to the CertEnroll generator inside Tools > Fiddler Options > HTTPS > Certificates Generated By.
The text below is still applicable for apps which implement pinning.
TLS1.0 is perhaps the best-supported HTTPS protocol in Fiddler. You haven't shown what's in Fiddler's Web Sessions list or Log tab in the event of the failure, but my guess is that the Web Sessions list probably shows just a CONNECT and the Log tab has something like:
!SecureClientPipeDirect failed: System.IO.IOException Authentication
failed because the remote party has closed the transport stream. for
pipe (CN=*.kayak.com, O=DO_NOT_TRUST, OU=Created by
http://www.fiddler2.com)
Is that correct? If so, the most likely explanation is that the Android app in question has enabled certificate pinning.
From the Fiddler book:
Certificate Pinning
A very small number of HTTPS client applications support a feature
known as “Certificate Pinning” whereby the client application is
hardcoded to accept only one specific certificate. Even if the
connection uses a certificate that chains to a root that is otherwise
fully-trusted by the operating system, such applications will refuse
to accept an unexpected certificate.
To date, some Twitter and
Dropbox apps include this feature, and Windows 8 Metro apps may opt-in
to requiring specific certificates rather than relying upon the
system’s Trusted Root store. Firefox’s automatic browser update
feature will silently fail when Fiddler is decrypting its traffic. The
Microsoft Security toolkit named EMET can enable pinning in any
application for certain “high-value” sites (including Windows Live).
The Chrome browser supports pinning, but it exempts locally-trusted
roots like Fiddler’s.
When a Certificate-Pinned application performs a
HTTPS handshake through a CONNECT tunnel to Fiddler, it will examine
the response’s certificate and refuse to send any further requests
when it discovers the Fiddler-generated certificate. Unfortunately,
there is no general-purpose workaround to resolve this; the best you
can do is to exempt that application’s traffic from decryption using
the HTTPS tab or by setting the x-no-decrypt Session flag on the
CONNECT tunnel. The flag will prevent Fiddler from decrypting the
traffic in the tunnel and it will flow through Fiddler uninterrupted.
A very small number of HTTPS client applications support a feature
known as “Certificate Pinning” whereby the client application is
hardcoded to accept only one specific certificate. Even if the
connection uses a certificate that chains to a root that is otherwise
fully-trusted by the operating system, such applications will refuse
to accept an unexpected certificate. To date, some Twitter and
Dropbox apps include this feature, and Windows 8 Metro apps may opt-in
to requiring specific certificates rather than relying upon the
system’s Trusted Root store. Firefox’s automatic browser update
feature will silently fail when Fiddler is decrypting its traffic. The
Microsoft Security toolkit named EMET can enable pinning in any
application for certain “high-value” sites (including Windows Live).
The Chrome browser supports pinning, but it exempts locally-trusted
roots like Fiddler’s. When a Certificate-Pinned application performs a
HTTPS handshake through a CONNECT tunnel to Fiddler, it will examine
the response’s certificate and refuse to send any further requests
when it discovers the Fiddler-generated certificate.
Unfortunately,
there is no general-purpose workaround to resolve this; the best you
can do is to exempt that application’s traffic from decryption using
the HTTPS tab or by setting the x-no-decrypt Session flag on the
CONNECT tunnel. The flag will prevent Fiddler from decrypting the
traffic in the tunnel and it will flow through Fiddler uninterrupted.
If you're very serious about circumventing pinning, you can jailbreak the device and use any of a number of 3rd party toolkits to disable the pinning code.
I want to monitor HTTPS traffic from my application to remote server. I am trying to follow this instruction and it works for HTTP (without s), but not for HTTPS.
What is wrong? Should I write some custom code in my application to use https-proxy ?
The easiest way to do this is to use CharlesProxy to proxy your device or emulator traffic for you. The only extra step you need to do is to install the CharlesProxy SSL certificate on your device/emulator which is very straight forward:
Download the certificate from Charles Proxy (it's in their help menu) and place it on your device, then install via security settings on your device.
You then configure your device or emulators network connection to use a manual proxy and set it to the Charles Proxy address and port. Enable SSL proxying and your SSL connections will be securely routed end-to-end via Charles and Charles will be able to show you the content of requests and responses in the clear.
I'm using WireShark for sniffing, it allow you to monitor and filter raw data. But because you using https and all transactions encrypted i suppose it can't help you. May be you can switch from https to http for debug, and later when all will be works fine change protocol back to https
Do you mean you can't see the traffic at all or do you get it encrypted? Is this a web application or native application? which Android version are you using? phone or emulator?
Normally, if you set up the proxy properly, you will get the traffic, but encrypted so you can't read it. In order to see the actual content in Fiddler you would need your device to trust Fiddler's root certificate (used to create fake certificates on the fly). See this:
http://www.fiddler2.com/fiddler/help/httpsdecryption.asp
Unfortunately, I have not found a way to add root certificates to an android device other than
rooting it and replacing the certificate store (like this)
https means http secure, so it obviously can't be sniffed so easily. what would be the point if it would be the same unsecure thing as normal http?
you have to learn a bit more about secure network comunications. or, long story short, at least you will have to learn how to use a specilly devised http proxy like charles http://www.charlesproxy.com/documentation/welcome/ so you will be able to monitor you own https traffic in a clear form.