When I want to view SSL traffic which is being proxied through Charles I need to have an SSL certificate from Charles installed on the smartphone. Why it's needed and how it's possible for Charles to decrypt and then encrypt again the data routed through it?
I imagine it in the way that if some smartphone app uses SSL then:
the data is encrypted (by the app? by the OS' network layers?) then
the encrypted data is sent to the world and
the encrypted data is intercepted by Charles
Charles gets the encrypted data and what now?
How does it know how to decrypt the encrypted data? And how does it know how to encrypt the decrypted data again to send it finally to where it was originally targeted?
Ok, what I'm going to post here is just how I think SSL proxying with Charles works, but I don't have any solid base to ensure my answer is correct. In fact, it would be great if someone from the Charles Proxy team could help us on this.
The point is that when your application performs an HTTPS request to a site firstly it has to go through the Charles Proxy (don't forget it is a proxy!). At this moment, Charles connects to the https site by using the site's public key to encrypt and decrypt data, as if it was a regular browser or application.
So at this moment Charles has the response from your https request unencrypted, and this must be passed to your application, but your application is expecting encrypted data, so Charles has to encrypt it again so your application (i.e.: your browser) doesn't complain about an uncrypted https response. To do so, Charles uses his own certificate (public + private key pair), encrypts your data and sends it back to your application.
Finally, your application receives this data encrypted by Charles. Your application won't know how to decrypt it unless you give him the public key of the certificate (this is done by "installing" the certificate on your application/browser/android device/etc.).
This is how I think all the SSL proxying works with Charles.
It would be really great if someone could contribute to this answer!
I'm a creator of Proxyman, which is a macOS web debugging proxy like Charles Proxy, so I might have the insight and experience to answer your questions.
When you start Proxyman/Charles, it will open a server at IP=127.0.0.1, port 9090, or 8080. It also overrides the HTTP/HTTPS Proxy in Network -> Wifi -> Advanced -> Proxies tab.
It's essential to make sure all traffic will go through Proxyman/Charles server.
If it's an iPhone, you have to manually set a Proxy config in the Setting app. Guideline: https://docs.proxyman.io/debug-devices/ios-device
As soon as the client (macOS) or iOS app (iOS devices) makes an HTTPS request with a Proxy Config. The first request is a CONNECT request to Proxyman/Charles Server.
Proxyman accepts it and returns 200 OK Status.
Your app and Proxyman server would perform SSL Handshake.
In order to make it successful, you have to install and trust Proxyman/Charles CA Certificate, which is a self-signed certificate.
For macOS: Install & trust in System Keychain.app
For iOS: You have to visit http://proxy.man/ssl (or http://chls.pro/ssl)
4.1. Proxyman extracts the Host key in the previous CONNECT request and starts fetching the actual certificate server.
4.2. Proxyman parses the certificate and gets all certificate information, such as Org, Name, Common Name, Subject Alternative Name (DNS and IP Address), etc
From what I know, Charles Proxy doesn't perform this step. It just generates a basic leaf certificate. Some clients can detect the missing data in the leaf certificate and reject it.
4.3. Proxyman starts generating a leaf certificate by using Proxyman CA Certificate. It's important to construct the same certificate that has the same information from the server.
If it's a macOS/iOS app, make sure the certificate is complied with a new security requirement from Apple. Otherwise, it might be rejected.
This step can be implemented by using OpenSSL or BoringSSL library.
4.4. Proxyman server sends the leaf certificate back to the client.
4.5. The client will verify the leaf certificate and it's passed. Because the CA Certificate is trusted in their system and it passes some basic evaluation, such as:
Common Name and DNS name must equal the Host
It's not expired
Use acceptance encrypting algorithms
and more
At this point, the SSL Handshake is complete, and the client starts sending the HTTPS Request.
Proxyman now acts as a Server, so Proxyman can decrypt the read the HTTPS Request in plain text.
Proxyman now acts as a Client and connects to the server. Then start sending a request. This step is really easy because the server's certificate is signed by a well-known CA certificate.
As soon as Proxyman receives a Response, Proxyman reads it and sends it back to the client.
The client receives the Response and works as normal.
Since Proxyman can see the Request/Response in plain text, it can display on the app or performs few debugging tools to modify the data, such as Map Local, Scripting, Breakpoint, etc.
It's a long answer and I hope that can answer your concern.
What Charles does is basically a Man-in-the-middle attack. That's why you have to install it's certificate on the device you are using.
Usually when a device and server communicate via ssl/tls, the request and responses are encrypted using the server's certificate. This means that someone who intercepts the traffic can not read it since it's encrypted with the server's certificate. But since you installed Charles' certificate (meaning your device trusts it) on your device and you set it up to communicate to the server via the proxy, Charles is able to decrypt the traffic.
The steps would look like this:
client and server do a key exchange (however since Charles is in the middle, they exchange keys with Charles)
client sends an encrypted request to the server (however it uses the key from Charles)
Charles intercepts the request
Charles decrypts the request and that's what you see
Charles encrypts the request with the server's key
Charles sends it to the server
And the same happens for responses.
Related
I would like to send a request to a webserver. It should be secure against Man-In-The-Middle-Attacks. Therefore I have created a SSL certificate on the webserver (https). As a response I would get a random String. This works well.
But how can I secure the response to protect it against MITM-Attacks? How should the response be sent from the webserver to Android and what do I have to configure on Android? Do I have to buy another trusted certificate?
In a typical as-called 1-way TLS (a.k.a "server certificate authentication") setup a server would listen requests from basically any client, but the clients would only trust the server with a valid certificate. When the client thinks it can trust the received certificate, the communication channel can be opened and it will be encrypted both ways.
As-called 2-way TLS (a.k.a "client certificate authentication" or "mutual authentication") setup means that both ends present a certificate to the other end. In other words your server would trust only certain clients (the ones that present a valid certificate). This would have no impact on your protection against MITM. You would just be limiting the pool of trusted clients.
So as a conclusion - if your only concern is to protect your communication against MITM-attacks, 1-way TLS is fine.
TLS v1.2: https://www.rfc-editor.org/rfc/rfc5246
I'm setting up a server which an android app and an iPhone app will connect to. And I'm wondering what type of security is more secure for sending/requesting data?
Currently I generate a HMAC-SHA256 of the content I'm sending to the server in the header to verify its integrity.
But I'm wondering if its more secure to use a https connection instead? If I use https, could I skip the HMAC?
I would like to know the differences in security, which is more secure?
And also, if I'm using either is it better to use both for an extra layer of security?
Quick answer to your questions: SSL if used properly should give you more security guarantees than HMAC. So, usually SSL can be used in a way that removes the need for HMAC.
HMAC provides integrity as well as authenticity. Assuming the client and the server use pre-shared symmetric keys to calculate the HMACs, one side can be sure that the device on the other end has the secret key. This provides authenticity of both server and client.
What is missing in this picture (with just HMAC) is confidentiality. What is the nature of data exchanged between the server and client? Is there any sensitive user data being transferred during the communication that you don't want a man-in-the-middle to see? If so, then you may want to use SSL.
SSL gives you confidentiality (among other things). Meaning that you can be sure that you have a secured end-to-end connection and no man-in-the-middle can see what data is being exchanged between the server and client. However, common SSL usage does not include client machine authentication. Fro example, your web browser checks for Paypal's authenticity when you go to their https webpage. But the Paypal server does not ask your browser to send any certificate from your side.
Since you are comparing SSL with HMAC, I am assuming you care about authenticity of both sides. So, use SSL with both server and client authentication. This basically means that both of them would ask for each other's certificates and check different aspects of the certificates (i.e. common name, certificate issuer etc.). You can create your own certificate issuer to sign these certificates.
If you are making an app for AppStore or Google Play that users can simply install and start using, you may want to think through how the client side certificates will be generated, signed or who will sign them. You can remove the need for client side certificate (and signing) by adopting a model similar to GitHub's, where the user manually informs the server of trusted public keys to authenticate devices. But you can probably see how this process might not be user friendly.
I am using k9mail(Android) open source for my email client application. The objective is, The users only access my email server with SSL security. so that I need to install SSL certificate in my email client. What is the best way I want to install SSL certificate at my client side?
(i.e) Install it from my server when the client communicate with server (or) Install it in my client When I develop application.
Can you give your suggestions?
Short answer:
Install/include it in the client app when you develop it. You should not install it from the server when the client communicates with it - it is not a correct approach (security wise).
Long answer:
The server certificate must be transferred to the client in a secure way. That is to avoid an attacker to replace it with his own certificate so that later on it will pretend that he, the attacker, is the mail server (man-in-the-middle attack).
If you just download the certificate on your client when the client communicates with the server we cannot prove that the entity you think is your server is really your server unless you have already established a secure (encrypted/authenticated) channel with it (in which case you wouldn't need your certificates anyway if you would have that secure connection already).
So you will have to generate your server certificate before hand and include it in the application. The alternative way would be to transfer it via a physical medium or other secure channel (which doesn't really make sense..)
I have client/server data passing all working correctly. Text, Images, etc. My users create blog-type posts on their android device, and upload to my server.. All is done using HTTP Multipart and Input/Output Streams. My issue is - How do I know the client is actually my app and not some script/other hacker app?
I want to avoid abuse scenarios.
Malicious user A creates a PC script that sends the appropriate form data to my server and is able to spam the server, creating 1000s of malicious posts.
Malicious user B creates a simple Android App that sends the appropriate form data to my server and he is able to spam the server.
Malicious user C signs up to my service, Has a valid account and password, and he spams the server using a PC script or Android App.
One idea I have is to force a wait period server side on frequent posts to prevent spam..
But beyond that, how can I check that the person sending data to my server is
An android device and
Is running my App to send form data and not another.
I want to avoid SSL as I don't want to register with Verisign, TRUST and go through all of that..
If it's only your client and your server, you can (and should) use SSL without purchasing anything. You control the server and the client, so each should only trust one certificate, the one belonging to the other and you don't need CAs for this purpose.
Here's the high-level approach. Create a self-signed server SSL certificate and deploy on your web server. You can use the keytool included with the Android SDK for this purpose. Then create a self-signed client and deploy that within your application in a custom keystore included in your application as a resource (keytool will generate this as well). Configure the server to require client-side SSL authentication and to only accept the client certificate you generated. Configure the client to use that client-side certificate to identify itself and only accept the one server-side certificate you installed on your server for that part of it.
A step-by-step for this is a much longer answer than is warranted here. I would suggest doing this in stages as there are resources on the web about how to deal with self-signed SSL certificate in Android, both server and client side. There is also a complete walk-through in my book, Application Security for the Android Platform, published by O'Reilly.
I finally got it all working, server and client two-way ssl authentication.
I used the instructions here to setup my own cert authority (ca) http://www.garex.net/apache/
I followed the commands there to generate my own ca, server and client files..
The big "GOTCHA" was that in the "create client certificate" section, the garex.net link uses a 1024 size client key. As it turns out, this was throwing the exception java.io.IOException: Wrong version of key store
To get around the above exception, I had to use only 512 sized keys.. This is done by NOT including the "1024" parameter to the openssl genrsa genkey command..
Finally I want to add a link to a tool I ended up using instead of Portecle.. I found the keytool gui program here of great help and easier to use than the portecle one - http://www.lazgosoftware.com/kse/index.html
This issue was a bit of a pain in the butt so I will keep an eye on this thread.. Feel free to reply if you run into any roadblocks..
You can use a captcha to solve this problem.
Before submitting the post, request the server for a captcha.
Server associates a random captcha image with a unique key, and sends the application both the captcha and the key.
Show the captcha image to the user.
Send the post, the letters entered by the user and the unique key in your http request.
Server verifies the captcha letters based on the unique key.
If captcha verification succeeded you add the post, otherwise you don't.
This should solve all the scenarios.
I am developing an Android app which need to consume .Net webservices over SSL which I have no experience in. Now I am looking for some guidance and explanation on SSL handshake and certificates.
Note: the server is using IP address and NOT domain name. It is an intranet application.
So far I have created a certificate(called self-signed?) in web server from IIS 7.
To consume it from Android app, I found two ways of doing it :
1). Embedded the certificate in the app (Which certificate? How do I get it?)
2). Trust all the certificates ( ppl said there is security issue with this approach, could you elaborate more? Does it still do the handshake?)
CERTIFICATES:
How many type of certificates are there in the handshake and what are they?
Does self-signed certificate have root certificate? If yes, how can i get them?
Is it possible to move/copy the self-signed certificate from one server to another?
HANDSHAKE:
First of all, is this process correct?
The SSL handshake process(copied from a website) is described below:
The client initiates the SSL handshake process by sending a URL
starting with the following: https:// to the server.
The client initially sends the Web server a list of each encryption
algorithm which it supports. Algorithms supported by SSL include RC4
and Data Encryption Standard (DES). The client also sends the server
its random challenge string which will be utilized later in the
process.
Will the embedded cert be sent in here?
The Web server next performs the following tasks:
Selects an encryption algorithm from the list of encryption
algorithms supported by, and received from the client.
Sends the client a copy of its server certificate.
Sends the client its random challenge string
The client utilizes the copy of the server certificate received from
the server to authenticate the identity of the server.
The client obtains the public key of the server from the server
certificate.
The client next generates a premaster secret. This is a different
random string which will in turn be utilized to generate the session
key for the SSL session. The client then encrypts a different value
called the premaster secret using the public key of the server, and
returns this encrypted value to the server. This is accompanied with
a keyed hash of the handshake messages, and a master key. The hash
is used to protect the messages exchanged in the handshake process.
The hash is generated from the former two random strings transmitted
between the server and the client.
What is a master key?
The server sends the client a keyed hash of all the handshake
messages exchanged between the two parties so far.
What is this keyed hash made from?
The server and the client then generate the session key from the
different random values and keys, and by applying a mathematical
calculation.
The session key is used as a shared secret key to encrypt and
decrypt data exchanged between the server and the client.
The session key is discarded when the SSL session either times-out or is terminated.
I'll try to answer to the best of my knowledge here
Embedded the certificate in the app (Which certificate? How do I get it?)
This the certificate identifying the client's/app identity. You can either get it through CA or self signed. This certificate will be used by the server to verify the client's/app identity
Trust all the certificates ( ppl said there is security issue with this approach, could you elaborate more? Does it still do the handshake?)
It still does the handshake but it doesn't do the certificate validation which is dangerous unless you are connecting internally (which seems you are). Trusting all certificate means an entity can claim as someone who they are not and thus could obtain confidential information from the users.
How many type of certificates are there in the handshake and what are they? In handshake you have the server's certificate and optionally the client certificate (for two factors authentication)
Does self-signed certificate have root certificate? If yes, how can i get them? Root certificate as far as I know means the ones that identifies by CA itself and thus it has no else to sign it. As your identity can still be verified and needs to be signed by CA, yours would not be classified as root certificate
Is it possible to move/copy the self-signed certificate from one server to another? The short answer is yes though the procedures from one platform to the others are different. Check [this link)(http://www.sslshopper.com/how-to-move-or-copy-an-ssl-certificate-from-one-server-to-another.html), it has instructions to copy certificate for few platforms
Will the embedded cert be sent in here? No, the embedded (client's) certificate is sent after validation of the server's identify is complete
What is a master key? Master key is the key that is used to derived the session key for later communication. It is also used to hash the messages and to verify authenticity of the messages in the next set of stages
What is this keyed hash made from? It's made from the master key sent by the client. In order to verify all messages, the server sent all messages that have been passed and hashed it with the master key. The client will hashed its messages as well with the same key and then compared with the data sent by the server. Only when the hash matches then we could be sure we are still communicating with the same server