Analyze Android app's data transmission (both HTTP and HTTPS) using MITM (Man-in-the-middle)

Ever wondered, what your android app is secretly sending to its servers? This is my attempt, at trying to intercept and decrypted the transmission by an android app. To understand what’s happening on my device.

For this, we do a Man-In-The-Middle (MITM), which means that rather than having my phone communicate directly with the internet, I will make it go through my laptop and watch the data which goes/comes. We will use Charles Proxy (trial version), this will be used to intercept the traffic. In order to do this, we will install this software on laptop (and route the network data from phone through this software).

To analyze just HTTP traffic, a non-rooted phone works. In case you wish to analyze the encrypted (HTTPS) traffic, we would need a rooted phone.

We will be showing all the steps on a macOS machine. Similar steps can be found out for other operating systems.

Intercept HTTP Traffic (Unencrypted)

Install softwares on laptop

  1. https://www.charlesproxy.com/download/. We will use the unpaid version, which has limited session size of 30 minutes.

Configure Proxy Settings on android

  1. Go to settings
  2. Tap Wi-Fi
  3. (for Redmi’s OS), click on the arrow button next to the connected Wireless
  4. Scroll down to Proxy and tap Manual
  5. Fill in the IP Address in Hostname, and Port Number in Port.
  6. NOTE: On your Mac, press alt and click on the wireless Icon, to know your laptop’s IP address. You can find required port number in Charles App. In menu bar, Proxy > Proxy Settings. If you are unable to find IP and port number, see: https://community.tealiumiq.com/t5/Tealium-for-Android/Setting-up-Charles-to-Proxy-your-Android-Device/ta-p/5121#configure_charles

Simply use the app whose communication you wish to interpret, and you will be able to see the transmitted data in your laptop (on Charles)

Intercept+Decrypt HTTPS Traffic (Encrypted)

Install softwares on laptop

  1. https://www.charlesproxy.com/download/. We will use the unpaid version, which has limited session size of 30 minutes.
  2. Install brew if you do not already have it; https://brew.sh/ (just run /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" in Terminal)
  3. brew cask install android-platform-tools, in order to install adb
  4. brew install apktool (for decompile-recompile)

Configure Proxy Settings on android

  1. Go to settings
  2. Tap Wi-Fi
  3. (for Redmi’s OS), click on the arrow button next to the connected Wireless
  4. Scroll down to Proxy and tap Manual
  5. Fill in the IP Address in Hostname, and Port Number in Port.
  6. NOTE: On your Mac, press alt and click on the wireless Icon, to know your laptop’s IP address. You can find required port number in Charles App. In menu bar, Proxy > Proxy Settings. If you are unable to find IP and port number, see: https://community.tealiumiq.com/t5/Tealium-for-Android/Setting-up-Charles-to-Proxy-your-Android-Device/ta-p/5121#configure_charles

Setting Charles to use SSL Proxying on all locations

Go to Proxy > SSL Proxying Settings, and add *.*. For more details, see: https://community.tealiumiq.com/t5/Tealium-for-Android/Setting-up-Charles-to-Proxy-your-Android-Device/ta-p/5121#configure_charles

Rooting your android phone

NOTE: Do it only if you are okay, if your phone gets bricked (the OS can get corrupted), and you will need to spend some time to fix it. I do not take ANY Responsibility, to what these steps might result it. This is just a summarization of the different blogs/websites that I had to follow, to achieve the results I needed.

I used a Redmi Note 4 for this task. YouTube video describing the process

  1. Unlock the bootloader. (Youtube video)
    1. Make sure you have an account signed in with MIUI (Settings > Mi Account)
    2. Enable Developer options Settings > About phone > MIUI version, tap on MIUI version for 7-10 times continously, which will give you a pop up saying that developer options are now enabled
    3. Settings > Additional settings (under SYSTEM SETTINGS category) > Developer options > OEM unlocking
    4. Go to https://en.miui.com/unlock/, and click on Unlock Now.
    5. It leads to https://en.miui.com/unlock/download_en.html, from where you can install a zip containing Mi Unlock app (needs to be run on a Windows machine), which is used to unlock bootloader. Sign in to your MI Account in this app
    6. Shut down your phone manually, and hold Volume down key and Power button to enter Fastboot mode
    7. Connect your phone to PC using USB cable and click “Unlock”. NOTE: A lot of blogs/websites mention that you need to fill up a form and wait for your request to be approved, but in my case I did not get any such thing and my bootloader was unlocked immediately (it might be the case that they have removed the review process now)
  2. Transfer the required files to your phone: Super SU, Lazy Flasher (these links are from YouTube video)
  3. Put phone in Fastboot mode (Switch off your phone, and power on by keeping Power button + Volume Down button pressed)
  4. Connect the phone to laptop
  5. Ensure fastboot devices list an entry
  6. Flash TWRP Recovery. (for Redmi Note 4; I downloaded RedWolf, mentioned in forum.xda-developers). fastboot flash recovery <recovery-img-path>
  7. Restart phone with Power button + Volume Up button
  8. Install Super SU zip using TWRP (RedWolf)
  9. Install Lazy Flasher using TWRP (RedWolf)
  10. Reboot your phone
  11. Your phone should be rooted now! You will be able to find a SuperSU app installed.
  12. Now you can do adb shell and su, to access all the files on your phone through ADB, and much more! (when running the su command, you might have to authorize it from the SuperSU app on your phone)

Installing the certificate as root (requires rooted phone)

  1. Install Root Certificate Manager(ROOT) android app (from Google Play Store).
  2. Download the certificate, by visiting chls.pro/ssl (needs Charles to be running and wireless proxy configured on phone)
  3. Add the certificate using the app

You have two options now

1. Xposed Framework

You can install the Xposed Framework. There is a Youtube Video which describes this complete procress: https://www.youtube.com/watch?v=yJRlMmJjrhY

  1. Download the framework
    1. https://repo.xposed.info/module/de.robv.android.xposed.installer, which links to
    2. http://forum.xda-developers.com/showthread.php?t=3034811 (for Android 5 or higher), which links to
    3. https://dl-xda.xposed.info/framework/, choose the SDK corresponding to your android version. You can check this on https://source.android.com/setup/start/build-numbers (API Level number corresponding to your android version). In my case, it was SDK 24, corresponding to Android 7.0, which links to
    4. https://dl-xda.xposed.info/framework/sdk24/, searching on google for the phone architecture lead me to a post on xda-developers forum, which suggested that Redmi Note 4, has arm64 architecture, which links to
    5. https://dl-xda.xposed.info/framework/sdk24/arm64/, download the latest version
    6. which in my case, is https://dl-xda.xposed.info/framework/sdk24/arm64/xposed-v89-sdk24-arm64.zip
  2. Transfer this zip to your phone
  3. Reboot phone in recovery mode, and install this zip, using TWRP (or similar)
  4. Now you need to install SSL Unpinning module for Xposed Framework. (https://repo.xposed.info/module/mobi.acpm.sslunpinning which links to, https://dl-xda.xposed.info/modules/mobi.acpm.sslunpinning_v2_37f44f.apk).
  5. Install the downloaded APK
  6. Open Xposed Framework app, and select Modules from the left side bar. (Your phone might need a restart for this module to appear)
  7. Enable (tick) SSL Unpinning module, and restart your phone
  8. Open the SSL Unpinning app, and click on a listed app, to unpin SSL certificate for it.
  9. Now you should be able to view the app’s encrypted data (in unencrypted form) in Charles. Though this method might not work for some applications.

[OR] 2. Recompile the APK

Decompile APK, add some parameters, recompile and install the APK. Blog describes the process in a detailed manner, I am adding the problems+solutions I had to use on top of what is mentioned there.

  1. Get the installed apk from your Android device
    1. adb devices ensure that your device shows up (try other USB ports if the phone does not show up)
    2. adb shell pm list packages -f app-name to get the path of the application
    3. You essentially need to run adb pull /data/app/bazingaa.nusbuses-1/base.apk (replacing your app name instead), but this did not work for me, as the directory /data/app/bazingaa.nusbuses-1/base.apk is only allowed access for root user. To overcome this, I used a process described at: https://android.stackexchange.com/a/129665/246838
    4. adb shell
    5. su
    6. cp /data/path/of/file/copyme /data/local/tmp
    7. chown shell.shell /data/local/tmp/copyme
    8. exit
    9. exit
    10. adb pull /data/local/tmp/copyme /destination/copyme
  2. Decompile the APK, with command apktool d -f base.apk (rename your APK to base.apk)
  3. android:networkSecurityConfig="@xml/network_security_config" and android:debuggable="true" need to be added under application tag in AndroidManifest.xml file. (for more details check https://medium.com/@ferrygunawan/debug-ssl-traffic-with-charles-proxy-872aedb228d)
  4. Add/Modify res/xml/network_security_config.xml, with content: 
     <?xml version="1.0" encoding="utf-8"?>
 <network-security-config>
     <debug-overrides>
         <trust-anchors>
             <!-- Trust user added CAs while debuggable only -->
             <certificates src="user" />
         </trust-anchors>
     </debug-overrides>
 </network-security-config>
  5. Rebuild the APK apktool b base
  6. If there is no error, you will get the base.apk under base/dist directory.
  7. You would need to sign the APK (reference)
    1. keytool -genkey -v -keystore my-release-key.keystore -alias alias_name -keyalg RSA -validity 10000
    2. Run jarsigner -verbose -keystore my-release-key.keystore <your-app-file> alias_name after replacing <your-app-file> with your file name, eg. jarsigner --verbose --keystore my-release-key.keystore base.apk alias_name NOTE: You may change alias_name and my-release-key.keystore, if you wish; but you need to keep it consistent across the two commands
  8. Uninstall the original app from your phone
  9. Install signed-APK that you created, using adb install base-signed.apk. You might need to enable apps to be installed via USB, under Developer options.

Extra; to look at the data that the app is storing locally

Ref

  1. adb shell
  2. su
  3. cd /data/data/
  4. ls, will show a list of all the app folders
  5. cd <appropriate-folder-name>, for example: cd com.android.bluetooth. This will take you inside the directory used to store all the app data and you can explore here, to find what you are looking for.

If you wish to copy this on your local system (for ease of viewing), you can use adb pull. For me, it did not let me directly pull the folder, as it requires su to view it. As a workaround, you can run the following commands:

  1. adb shell
  2. su
  3. cp /data/path/of/file/copyme /data/local/tmp
  4. chown shell.shell /data/local/tmp/copyme
  5. exit
  6. exit
  7. adb pull /data/local/tmp/copyme /destination/copyme
Written on April 16, 2020