Most app have some form of SSL pinning system in place which means that you have to perform additional work to allow the proxy to decrypt the HTTPS traffic.
I would say based on personal observation that the more scrape-worthy an app is, the more likely it has cert pinning. Rather obvious if you think about it, really. High value targets especially from big shops tend to have other measures like complex MACs that make scraping hell.
I’m sure most largely-worthless-to-scrape apps don’t employ cert pinning.
Recent example I encountered: TikTok web API has dynamically generated parameters X-Bogus, msToken and _signature (could be slightly wrong, it’s been a while) that are verified server-side. I haven’t reversed their mobile app so not sure if they also employ MACs there, but I’ve seen these from other apps in the past. And it’s harder when employed in an app; on the web you’ll be reversing (obfuscated) JavaScript in a readily available debugger, whereas for an app you’ll likely be reversing from disassembly.
>Caution: Certificate Pinning is not recommended for Android applications due to the high risk of future server configuration changes, such as changing to another Certificate Authority, rendering the application unable to connect to the server without receiving a client software update.
This actually applies to websites and browsers as well.
Every escape hatch in the certificate validation is also an additional avenue for attack. For example, using a DNS record to override certificate pins makes DNS cache poisoning much more valuable to the attacker.
I have worked with the site mentioned in the article’s API previously. I am not sure why they used the overhead of a "scraping framework" when it was just JSON they needed to parse.
Perhaps they're used to the Scrapy APIs and tools? I agree that an HTTP requests library would have been sufficient but maybe it was just easier for them to use the framework they're used to.
That is really nice. The last time I attempted scraping an app was using an android emulator (bluestacks), then using maybe Wireshark or Charles for getting the API endpoint. It didn't work for some reason though. I don't remember the exact error
and I am kinda skeptical about app scraping being this easy.
I recently did something similar with good results (I found the api endpoints I was interested in) using the official Android emulator and https://github.com/mitmproxy/mitmproxy
I did have to jump through some hoops with the emulator and pushing my own ssl cert to it's RO system partition. But it was a few commands and easy enough.
This is a great method. It's essentially creating a custom client for their server. Fiddler makes interception of encrypted traffic from apps a lot easier than the last time I tried this. Really nice.
