Intro?
This topic needs no introduction. As the web moves towards an "encript everything" approach, so does the need to be able to decipher said encription grows stronger.
The right tool for the job
The first tool that jumps to mind when sniffing traffic is Fiddler. Everyone uses Fiddler, right?
But I couldn't for the life of me get Fiddler 4 to handle the emaultor's SSL traffic correctly on my Windows 10 box. My SSL connections would keep timing out, no matter whatI did, and it seems I'm not the only one facing this problem .
I did however have much more success with Charles - a proxy tool very similar to Fiddler albeit not a free one. But a price tag of 50$ per license seems very reasonable to me.
Setting up the proxy
We begin by downloading and installing Charles. Charles is cross platform and works on linux, windows and MacOs.
Once installed, we need to set it up correctly for capturing the Android emulator's traffic.
We enable HTTP proxy by clicking Proxy -> Proxy settings... and ticking Enable transparent HTTP proxying:
Since we're only interested in capturing the emulator's traffic and not the host computer's, let's turn off implicit traffic capturing, and only capture traffic that goes through the proxy explicitly. On the Windows tab we uncheck all of the boxes:
Setting up the emulator
Let's first verify that the emulator is able to reach the Charles proxy service correctly. We open the emulator's browser and point it to your Window's box address, on port 8888. You should see Charle's poorly designed welcome screen. If not - it's time for the usual debugging of a connection issue.
The average tutorial would now have you route your emulator traffic to the proxy by editing the proxy settings in the emulator's access point. But this only affects browser traffic and we want all traffic to go through the proxy, including application traffic. Therefore we need to add the proxy settings to the emulator's launch command.
Unfortunately, this is no longer possible from within Android Studio. So we would have to resort to running the emulator from the command line.
We list the available emulators:
C:\Users\Nitz>emulator -list-avds
Nexus_5X_API_22
Nexus_5X_API_22_64bit
And we launch our selected emulator while adding the -http-proxy {WINDOWS_BOX_IP}:{CHARLES_PORT}
switch. Default Charles port is 8888:
C:\Users\Nitz>emulator -avd Nexus_5X_API_22 -http-proxy 192.168.1.13:8888
Hax is enabled
Hax ram_size 0x60000000
HAX is working and emulator runs in fast virt mode.
adb_server_notify: Failed to establish connection to ADB server
emulator: Listening for console connections on port: 5554
emulator: Serial number of this emulator (for ADB): emulator-5554
While the emulator is booting, we can already see some traffic being caught by Charles (if not, press the Record button on the top. It's the second button from the left):
There are great tutorials out there on the subject of analyzing and manipulating traffic using Charles Proxy (link, link, link), so we won't dive into the subject in this one. Also - the main screen is very easy to understand as it is. You can see that my emulator did some requests to http://connectivitycheck.android.com/generate_204, and also some HTTPS requests which we can not decipher for now.
Let's have some fun. Here is me browsing the BBC news like a respectable gentleman:
Now let's try an HTTPS site: Wired.com. Notice that other than the root / request, all other requests are encrypted:
What about apps? Let's try Google Maps. I've launched it and navigated to a country named USA:
Let's fix this and begin sniffing ourselves some SSL traffic.
The story of a Man standing In The Middle
(Skip this chapter for the actual instructions)
Broadly speaking, a normal SSL connection is established by the client requesting a certificate from the server, verifying it against a list of approved certificate authorities, and if everything is OK then a client key is exchanged and then the encrypted communication begins (from here):
Once this is performed, we can not decipher the communication between the client and the server.
In order to defeat this process and be able to decipher the encrypted communication, Charles Proxy stands in the middle of the exchange. The client does a handshake with the proxy using a certificate issued by the proxy. Then the proxy impersonates the client and does another handshake with the server, this time with the server's certificate.
Any request from the client is first deciphered by the proxy, analyzed, and then encrypted again to be sent to the server and vice versa (from here):
The Charles Proxy "stands" in the middle of the communication. This is why it's called a Man In The Middle attack. But - in order to achieve this, the client needs to authorize Charles' SSL certificate. No respectable certificate authority would authorize a certificate for a proxy, so Charles provides a self signed certificate. We need to fool our emulator into thinking it's a valid one.
Setting up Charles for SSL capturing
First we enable SSL proxy. Proxy -> SSL Proxying Settings... and then we tick Enable SSL Proxying. We also want this enabled for all sites, so click on Add, leave both Host and Port empty and click OK:
We also head out to the Client Certificate tab and Add a new empty Host and Port entry:
We OK everything and Charles is finally fully configured for both HTTP and HTTPS.
Enabling SSL deciphering
The way any device verifies an SSL certificate is by testing it against Root Ceritificate, which is a public key of a known certificate authority.
So we need to install Charles Proxy's root certificate on the emulator. Open the emulator's browser and point it to chls.pro/ssl (the emulator has to still use Charles Proxy as a proxy). A prompt to install and name the new certificate should appear:
After installing the certificate you might be prompted to set up a lock screen PIN for your emulator. This is an unfortunate necessity in enabling credential storage on most Android devices.
You should now see a toast stating that Charles Proxy was installed, and a prompt on the notification bar would warn you that Network might be monitored.
Testing SSL deciphering
Let's try that Wired site again:
Notice that we can now decipher the encrypted transactions.
Now let's try the Google Maps app:
Not only that we have the app's communication fully available to us, Charles Proxy has also detected that some of it was an image request and parsed it for us.
Why not Charles all the things?
Now that we have a working SSL proxy sniffer set up, we might ask why bother with other solutions such as tcpdump that was described in a previous post?
Well - there are couple of possible answers, which also touch on the limitations of using a proxy:
- The emulator only directs TCP traffic to the proxy. tcpdump on the other hand handles UDP as well.
- The tcpdump approach is very good at handling large quantities of past data. The proxy approach is a realtime solution for capturing specific transactions.
- You do not always have access to a proxy machine, or not always able to configure a device to go through a proxy.
What's next
On the next few posts we'll handle SSL sniffing on real devices, iOS devices and emulators and Windows machines.