I am trying to debug an app on the Google Playstore so that I can see the requests that it is sending out. I've set up Charles Proxy on my computer to help me do so, but I am having issues with viewing requests sent out with https. My steps are listed below.
My Computer:
Manjaro Linux
My Phone:
Google Pixel 4a
Android 11
Not rooted
My Settings:
Using Charles v4.6.1
Proxy Settings
SSL Proxying Settings
My Access Control Setting's contain my phone's IP.
My Phone Settings:
Proxy Settings
Certificate
Here are my steps to allow the app use SSL through charles:
Download the app from the Google Playstore.
Find the apk and transfer it to my PC, and then uninstall the app through the Google Playstore.
Decompile the apk using Apktool from XDA Forums.
According to this and this, I need add the following lines below inside <network-security-config> inside the network security xml.
<debug-overrides>
<trust-anchors>
<certificates src="user"/>
</trust-anchors>
</debug-overrides>
The AndroidManifest.xml already follows the required changes, so I don't make any changes there.
I compile the decompiled contents into an apk.
I sign the built apk with Uber Apk Signer.
I transfer the built and signed apk to my phone, and install it.
I do not get any errors at any step though this list.
Here is a screenshot of my view in Charles.
Thank you in advance, and please let me know if you need any more details.
Your network config there is within <debug-overrides>, which only applies for debug builds. Did you build the application in debug mode, or for production? If you don't build in debug mode then that config won't apply.
You probably want to use <base-config> instead, which applies to all builds, not just debug builds. There's a full example here: https://httptoolkit.tech/docs/guides/android/#if-you-dont-have-a-custom-network-security-config.
If that doesn't work, then it's likely that there's some certificate pinning in place in the application code itself, independent of the network security settings. To fix that you'll need to manually edit the code itself. You can also try using https://github.com/shroudedcode/apk-mitm which has a selection of automated patches that disable many common manual pinning implementations for you.
Related
Please find the steps I followed when recording the mobile device traffic
Connect the Jmeter(version 5.5) installed lap top and the Mobile device (Samsung galaxy Android 12) in same wifi network (same subnet)
Setup the lap top's ip address(ipconfg command) in mobile wifi settings as manual proxy.
Setup the Jmeter setup port in mobile wifi settings as manual proxy - port.
Note - Traffic is allowed through mentioned port via Firewall (inbound rule)
click start button in jmeter's HTTP(S) Test Script Recorder.
Import the jmeter CA certificate (generated in jmeter bin folder) in to mobile device.
Install it through Settings >> Biometrics and security >> Other security settings >> Install from device storage >> CA certificate >> Accept the warning and install
Login to browser from mobile and search any keyword(default samsung browser, cleared browser cache)
Observe the results recording in jmeter result tree.
Observation - No any traffic from mobile is recorded.
below logs are recorded only in jmeter console
Please help to know the reason how this issue need to be resolved.
I don't think these steps are sufficient for Android 12, you need to additionally follow the steps from Network security configuration page of Android documentation in order to instruct your application to trust JMeter's CA certificate.
add the next line to the application section of your app manifest
android:networkSecurityConfig="#xml/network_security_config"
create network_security_config.xml file under your app resources folder with the following content
<?xml version=“1.0” encoding="utf-8"?>
<network-security-config>
<debug-overrides>
<trust-anchors>
<!-- Trust user added CAs while debuggable only -->
<certificates src="user" />
</trust-anchors>
</debug-overrides>
</network-security-config>
build your app in debug mode
replace the app in the device with the debug .apk from the previous step
More information: Configure Android Devices for Proxy Recording
I found the solution by myself.
Solution:
Originally the network profile of my lap top was set as Public. see below.
Then I changed it to "Private". problem was resolved for me with that.
Thanks.
I have followed instructions on how to add fiddler certificate on android emulator, using both nox and memu emulators, as well as my android phone running marshmallow, I set the WiFi proxy to point to my PC over the local network, when I open a website using a web browser, things work fine, I receive the warning, I choose to proceed and the connection is successfully tunneled and decrypted using fiddler.
But, when I try to use other apps, connections fail! I see the tunnel connections, and then connection fails. My bet is, it's due to the invalid HTTPS certificate, so my question is, is there a way for me to install fiddler to the trusted authorities so connecting to it will go through without the warning? So I can finally debug HTTPS traffic from and to those apps.
I found similar questions here on SO, but none of them were exactly the same as mine, nor did they have the right answers, so I'm not sure if this question does in fact qualify as a duplicate.
Thanks
On modern Android devices using apps developed for target API Level 24 (Android 7) or higher sniffing traffic is not that simple anymore. The target API level of an app is defined it's AndroidManifest.xml file in the entry <uses-sdk android:targetSdkVersion="??"/>.
The main problem is that if you install the Fiddler root CA certificate in Android it is marked as user certificate (not system certificate). And unless explicitly configured in an app those user certificates are not trusted.
One of those rare apps that respect user CA certificates is Chrome. So using Chrome for testing if the proxy and the installed root CA certificate works is a bad idea, as it may only work in Chrome but not for apps.
Note that some apps further use certificate pinning (leaf or root CA pinning). Therefore even if the Fiddler root CA certificate is installed as system certificate the app won't trust this certificate as it fails on the certificate pinning.
Certificate pinning is also a web site feature, hence some sites save a certificate hash in the web browser cache that pins the site to a certain certificate. In such a case clearing the browser cache is usually removing those pinning data.
Rooted devices
If your device is rooted you can try to install the Fiddler root CA certificate as system certificate. The Mitmproxy documentation contains a how-to for manually installing the mitmproxy certificate.
If you have rooted the phone using Magisk, there is a Magisk module that seems to be able to install user certificates automatically as system certificates: https://github.com/NVISO-BE/MagiskTrustUserCerts
Alternatively you can install Magisk + Edxposed + TrustMeAlready Xposed module. This allows to disable certificate checking system wide - WARNING: this eliminates the security of SSL/TLS against active attacks, for all apps on the phone. Therefore only do this on a device you use just for hacking!
Also possible is installing and run Frida-Server on the device and hook into the app you are interested to modify the SSL/TLS certificate checking at run-time. AFAIK the Frida based framework Objection has some scripts to do so.
Non-rooted device
On a non-rooted device there is only the option to modify the application before you install it onto the device. Note that some apps will detect that they have been modified and will refuse to work.
To let the app trust user certificates you have to modify network_security_config.xml (see e.g. here) included in the app. You can use apktool to decompile/recompile the app. Don't forget to re-sign the recompiled/repackaged app e.g. using apksigner from Android SDK.
There are some tools available that automate the decompiling , modification and signing like apk-mitm.
There is also the possibility to modify an app by including the Frida gadget for Android into the app. This would allow to use Frida for this specific app on a non-rooted device.
I have followed instructions on how to add fiddler certificate on android emulator, using both nox and memu emulators, as well as my android phone running marshmallow, I set the WiFi proxy to point to my PC over the local network, when I open a website using a web browser, things work fine, I receive the warning, I choose to proceed and the connection is successfully tunneled and decrypted using fiddler.
But, when I try to use other apps, connections fail! I see the tunnel connections, and then connection fails. My bet is, it's due to the invalid HTTPS certificate, so my question is, is there a way for me to install fiddler to the trusted authorities so connecting to it will go through without the warning? So I can finally debug HTTPS traffic from and to those apps.
I found similar questions here on SO, but none of them were exactly the same as mine, nor did they have the right answers, so I'm not sure if this question does in fact qualify as a duplicate.
Thanks
On modern Android devices using apps developed for target API Level 24 (Android 7) or higher sniffing traffic is not that simple anymore. The target API level of an app is defined it's AndroidManifest.xml file in the entry <uses-sdk android:targetSdkVersion="??"/>.
The main problem is that if you install the Fiddler root CA certificate in Android it is marked as user certificate (not system certificate). And unless explicitly configured in an app those user certificates are not trusted.
One of those rare apps that respect user CA certificates is Chrome. So using Chrome for testing if the proxy and the installed root CA certificate works is a bad idea, as it may only work in Chrome but not for apps.
Note that some apps further use certificate pinning (leaf or root CA pinning). Therefore even if the Fiddler root CA certificate is installed as system certificate the app won't trust this certificate as it fails on the certificate pinning.
Certificate pinning is also a web site feature, hence some sites save a certificate hash in the web browser cache that pins the site to a certain certificate. In such a case clearing the browser cache is usually removing those pinning data.
Rooted devices
If your device is rooted you can try to install the Fiddler root CA certificate as system certificate. The Mitmproxy documentation contains a how-to for manually installing the mitmproxy certificate.
If you have rooted the phone using Magisk, there is a Magisk module that seems to be able to install user certificates automatically as system certificates: https://github.com/NVISO-BE/MagiskTrustUserCerts
Alternatively you can install Magisk + Edxposed + TrustMeAlready Xposed module. This allows to disable certificate checking system wide - WARNING: this eliminates the security of SSL/TLS against active attacks, for all apps on the phone. Therefore only do this on a device you use just for hacking!
Also possible is installing and run Frida-Server on the device and hook into the app you are interested to modify the SSL/TLS certificate checking at run-time. AFAIK the Frida based framework Objection has some scripts to do so.
Non-rooted device
On a non-rooted device there is only the option to modify the application before you install it onto the device. Note that some apps will detect that they have been modified and will refuse to work.
To let the app trust user certificates you have to modify network_security_config.xml (see e.g. here) included in the app. You can use apktool to decompile/recompile the app. Don't forget to re-sign the recompiled/repackaged app e.g. using apksigner from Android SDK.
There are some tools available that automate the decompiling , modification and signing like apk-mitm.
There is also the possibility to modify an app by including the Frida gadget for Android into the app. This would allow to use Frida for this specific app on a non-rooted device.
I'm using Nexus 5X device running Android 7. I failed to setup the proxy using the recent instructions from Charles Proxy documentation. I installed certificate, but wifi settings or nugat are extended. I can set charles as a certificate but that won't let me connect to that wifi (authentication problem). So I leave it as "don't check correctness". I don't even know if that's relevant.
The thing is when I try to sniff on my app https calls I see SSLHandshake: Received fatal alert: certificate_unknown
However if I run a web client using chrome on the same device - I can read calls to the same api.
The bottom line is it works for a browser but not for my app. I checked on other apps aswell. Same output.
The reason I ask here is because maybe I have to add some unsafe client to my retrofit api setup - hopefully not.
Here is how you need to configure you apk in order to make CharlesProxy work in Android 7.
Android
As of Android N, you need to add configuration to your app in order to have it trust the SSL certificates generated by Charles SSL Proxying. This means that you can only use SSL Proxying with apps that you control.
In order to configure your app to trust Charles, you need to add a Network Security Configuration File to your app. This file can override the system default, enabling your app to trust user installed CA certificates (e.g. the Charles Root Certificate). You can specify that this only applies in debug builds of your application, so that production builds use the default trust profile.
Add a file res/xml/network_security_config.xml to your app:
<network-security-config>
<debug-overrides>
<trust-anchors>
<!-- Trust user added CAs while debuggable only -->
<certificates src="user" />
</trust-anchors>
</debug-overrides>
</network-security-config>
Then add a reference to this file in your app's manifest, as follows:
<?xml version="1.0" encoding="utf-8"?>
<manifest ... >
<applicationandroid:networkSecurityConfig="#xml/network_security_config" ... >
...
</application>
</manifest>
Here you can find more details:
https://www.charlesproxy.com/documentation/using-charles/ssl-certificates/
In Charles, select Help -> SSL Proxying -> Install Charles Root Certificates in IOS Simulators, close the iOS Simulator and restart the simulator. It should work now.
I am connecting Mobile device with System via Proxy and capturing calls made in the mobile devices.
When "https" requests are captured, I am not able to get response from it.
Getting following error in System:-
No request was made. Possibly the SSL certificate was rejected.
Client Process: java
You may need to configure your browser or application to trust the Charles Root Certificate. See SSL Proxying in the Help menu.
Note: I have installed charles proxy ceritificate on system and also imported certificate in Mozilla Firefox. I am able to get response for "https" from Mozilla browser on System
I have loaded charlesproxy.com/getssl on mobile. It says "Certificate is already installed on your device"
When you downloaded the certificate from charlesproxy.com/getssl, was your mobile device connected via the Charlesproxy instance that you're trying to set up?
The more recent versions of Charlesproxy create a custom certificate, so the SSL proxying setup will only work if your device was being proxied at the moment when you installed the certificate. If it wasn't, you may want to delete it and install it again while connected via Charles.
I also like to restart charles and reboot the mobile device after installing a certificate. Maybe it's superstitious, but I've had cases where the SSL calls only worked after the reboot.
Installing ssl certificate on the device will only help with browser ssl logs, to trace ssl data of apps you will have to do the following.
Add a file res/xml/network_security_config.xml to your app:
<network-security-config>
<debug-overrides>
<trust-anchors>
<!-- Trust user added CAs while debuggable only -->
<certificates src="user" />
</trust-anchors>
</debug-overrides>
</network-security-config>
Then add a reference to this file in your app's manifest, as follows:
<?xml version="1.0" encoding="utf-8"?>
<manifest ... >
<application android:networkSecurityConfig="#xml/network_security_config" ... >
...
</application>
</manifest>
Source: https://www.charlesproxy.com/documentation/using-charles/ssl-certificates/
For what it's worth I always had problems connecting my Personal Android phone to charles proxy despite lots of troubleshooting - I would always get errors that my network had or the error in the OP.
I went into my phone's User Credentials System setting. I had 20 or so charles certs from previous attempts and other machines i've installed over the past two years - i deleted them all and things started working again. I hope that helps someone.