I have a Squid proxy installed with HTTPS/SSL. When I install the generated certificate in Firefox and set the proxy, everything looks good -- that is, I can see all the HTTPS requests the Squid log files.
However, when I install the same certificate in my Android phone (and change the proxy setting), it seems to work only in the browser. I first change the proxy setting and pages didn't load, then I installed the certificate and all was good.
All other app I've tried yield in errors due to connection problems. I can see the CONNECT request in the Squid logs but no other requests (e.g., GET, POST). I know that some app completely ignore the system proxy settings, but many use them.
The odd thing is that everything was working on an old Android 6.0 phone. Here the same apps where working perfectly fine using the proxy and relying on HTTPS requests. Now I have a newer Android 9.0 phone and all apps so far fail. The apps obviously use the proxy (seeing the CONNECT entries in the logs) but do not use/acknowledge the installed certificate.
I currently see the new phone or the new Android version as cause for the problem since it worked before with an older phone and Android version. How can I best track this issue down?
"Since Android 7, apps ignore user provided certificates, unless they are configured to use them. As most applications do not explicitly opt in to use user certificates [...] we need to place our CA certificate in the system certificate store"
https://docs.mitmproxy.org/stable/howto-install-system-trusted-ca-android/
However, this seems not to be easily done...
Android connects to specific domains to verify the connection.
Add this bit to the the squid.conf:
# Mobile
acl google-servers dstdom_regex "/etc/squid/google.txt"
always_direct allow google-servers
next, create this file /etc/squid/google.txt with the following content:
(^|\.)android\.clients\.google\.com$
(^|\.)ggpht\.com$
(^|\.)google\.com$
(^|\.)www\.googleapis\.com$
(^|\.)gstatic\.com$
(^|\.)gvt1\.com$
(^|\.)1e100\.net$
(^|\.)google\.co\.uk$
to activate, run:
./squid -k reconfigure
Related
Could you suggest how to add a self-signed certificate on Android (MIUI 12, Android 10).
There is a mobile app for Android, and a backend written in symfony. We have a bug that is very difficult to catch, and it is not clear whether it is a bug in the app or a problem in the backend. Currently, it is not known which endpoints in which order the mobile app calls. When I test my local backend through postman, everything works as expected (there is no bug). When I test on Android (which is directed to the stage), I can see a bug. Hence, it was decided to build the application directed to local backend by replacing the api-url with the hostname of my laptop. Then the requests would go over the local network to the local backend, where it would be possible to debug what exactly is happening. But once we've built the app, it turned out that it's not that simple to send requests to the local backend since a self-signed certificate is used. I tried to add this certificate to the android config, but nothing changed. Though, certificate shows up in the list. Maybe I made some mistakes when converting that certificate (because xiaomi doesn't allow to import a .crt file), or maybe it's backend that is configured incorrectly.
Can anyone who has encountered such a problem tell me how to set the environment up correctly. Maybe there is some guide on this topic.
PS. It is not an option to use Ngrok, because it will be necessary to distract the mobile team every time in order to build the application with the new url.
So I am trying to use Charles to test the analytics on this app. I've followed many tutorials but I can't seem to get it to work.
I've set up Charles properly, connected my Android phone (Google Pixel, Android 8) through the proxy, downloaded and installed the certificate. When I run the app I can see the tracking server, Charles filters the rest of them away, but the calls are blocked and instead I see a red "x" and this message:
"SS Handshake Received fatal alert:certificate_unknown"
The certificate is installed to the device and trusted I don't know what I'm doing wrong.
It's also important to note that the security config file has been added to the app, but I am still getting this result.
Error example:
It's also important to note that the security config file has been added to the app, but I am still getting this result.
Can you share your security config file? If that's configured to trust user-installed certificates then you shouldn't have this problem, so there may be something wrong there. Compare it with this example.
Alternatively, this might be traffic from a different app on the device. Changing the proxy settings will redirect all traffic from the device, and other unmodified apps won't trust your certificate, so will cause these exact errors.
I have installed and have been using charles proxy(4.5.6).
I am using Android 9.
Charles works on the Chrome pages (Instagram specifically):
but it does not work on the Instagram application:
and I always get that error when I am using the app.
Should I "Save Charles root certificate" and then install it on my phone? Also, why does Charles work on the Chrome but not on the Instagram app?
Instagram's app uses a technique known as SSL Certificate Pinning. Basically, this means they bundle information about the expected SSL certificate for the server into the app, and deny any connection that doesn't present that certificate. Because the certificate presented by Charles is not Instagram's certificate, the connection is denied by the Instagram app. In Chrome, it must rely on the browser's assessment of certificate trust, which will defer to certificates installed on the device - I assume you've already set up your device to trust the Charles Proxy SSL certificate, so because Chrome trusts it (because you told it to), the website works. There used to be a way to do key pinning in the browser, but it no longer works due to some issues that were discovered.
You can disable this certificate pinning in your Facebook account settings: https://www.facebook.com/whitehat/researcher-settings/
As for the details of their implementation, there are numerous articles online about how to disable the pinning in Instagram's app. I can't vouch for any of them in particular, as I haven't tried them, and I'm not sure how they would interact with the terms of service, but you could take a look at those for more information.
I have been working with Charles to monitor and debug network requests for my app and I would like to mention few points that would help you:
If you look at the documentation of Charles SSL Certificate for Android,
As of Android N, you need to add configuration to your app in order to have it trust the SSL certificates generated by Charles SSL Proxying. This means that you can only use SSL Proxying with apps that you control.
This restriction has been added by the Android framework itself to avoid exploits and hacks for more security. So, you cannot use Charles to montior or debug network requests of third party apps (in your case -Instagram) if you're having Android 7 (Nougat) or higher.
If you really want to do it, you can try connecting an Android phone with Android 6 (Marshmallow) or lower to see if it works for you.
If you want to debug your own app, your phone and the system running Charles should be on the same network and you need to setup proxy configuration in your phone's Wi-Fi settings.
First go to Charles -> Help -> SSL Proxying -> Install Charles Root Certificate on Mobile Device or Remote Browser
You'll see an info window like this:
Note the IP address mentioned in this window and go to your phone's Settings -> Wi-Fi -> Select your Wi-Fi network -> Edit
You'll find option called Proxy, it will be None by default, update it to Manual. You'll get two input fields for entering Hostname and Port, fill those details by looking at the IP mentioned in the Charles and Save it.
Another important point is, you'll need to install Charles Root Certificate on your phone. You can do it by visiting https://chls.pro/ssl from your phone's browser. It'll automatically download the certificate and will prompt you to install it. Make sure you're connected to Charles by following the 2nd step, otherwise it won't download the certificate automatically.
Once it's done, you're good to go!
There may be two reason for this
1) The instagram app is using network security configuration file which disabled the proxy servers
for more information read this website https://developer.android.com/training/articles/security-config
2) For checking payload of your own application you need to set proxy server
as https://community.tealiumiq.com/t5/Tealium-for-Android/Setting-up-Charles-to-Proxy-your-Android-Device/ta-p/5121
We have an important research project regarding mobile dating apps. One of the research points is also data being transferred to 3rd parties.
So, we had to decrypt HTTPS connections by doing a man-in-the-middle approach.
We used Fiddler as well as Mitmproxy. We also used Android emulator Bluestacks (rooted) as well as regular cell phones (Android < 6 (5.1) and Android v8 and v9). Installing root certificates worked fine.
However, we encountered a problem with one app "Grindr" (the world’s leading gay dating app).
We received "Grindr insecure connection" when starting the app. Deactivating the proxy, starting/using the app and then re-activated the proxy again,
we could capture HTTPS for just a short amount of time until this message has been displayed again. We assume that's an inner app code issue?
We downloaded different versions of app via apkmirror.com and used APK Tool to decompile the re-build the app and signed the app afterwards (e.g. APK signer, play store) and included some changes due to:
*Four Ways to Bypass Android SSL Verification and Certificate Pinning
https://blog.netspi.com/four-ways-bypass-android-ssl-verification-certificate-pinning/*
A wired thing: The original downloaded/mirrored APK could not be installed most of the time and if, after Grindr restarts, the app stopped. We tried different versions. (the lower versions worked better)
When there is no internet connection, the app opens properly and points to the login screen. I guess there must be a background check or URL callback (config.safedk.com?). However bypassing specific IP addresses didn't help.
Does anybody have experiences with this issue or any solution to overcome the "Grindr insecure connection" pop up?
I am having the most difficult time figuring out how to get a cell phone that will trust my local domain certificates without being very difficult or expensive.
Android devices tested up to 8.0 give a warning like: "Certificate authority installed by an unknown third party" or "Network May Be Monitored" on boot.
Is it possible to remove this message without rooting the phone? I need to set up always on VPN and SIP calling for the local pbx along with internal web sites. Android is the only offering with a native SIP client. But the warning causes too much trouble =/
The following worked for me in Android 8.
First transfer copy of the cert file to phone.
If using mkcert, the file location is found via CLI: mkcert -CAROOT
Then install the cert file in Android settings, the location of which varies per device and Android version.
In my phone it was in: Android Settings / General / Lock screen & security / Encryption & credentials / Install from storage
You might have to restart the phone. Also might have to click TRUST on the cert in Android settings.
Then enable Firefox secret settings by tapping multiple times on the Firefox logo in the About page, then in secret settings enable "Use third party CA certificates".
Voila!