Foursquare constantly tracking users' locations

(and how to audit your phone's application traffic yourself)

It might sound weird to accuse Foursquare of collecting location data since that is the whole point of the service, but Foursquare is overstepping its bounds by constantly keeping track of their users' every move (and more) -- even if they never open the app.

The Foursquare app contacts the service every ten minutes, providing a list of minute-by-minute locations (including timestamp and accuracy data), battery level, charging status, internet connectivity status and nearby wireless access points (complete with timestamp, MAC address and signal strength).

This is hardly news, though. Foursquare has proudly announced this new feature as the unnamed successor to Foursquare Radar. One important change is that the feature became opt-out (as opposed to Foursquare Radar, which was opt-in). The collected data enables them to provide users with push notifications of "tips, friends and interesting things etc. near you", as outlined in their privacy policy:

Also, Foursquare uses your mobile device’s ‘background location’ (formerly known as ‘Radar’) to provide the service, including to send you notifications of tips/friends/ interesting things etc. near you. If you have ‘background location’ turned on, the Foursquare app will, from time to time, tell us about your device’s location even if you are not directly interacting with the application.

They even addressed the concern that this might impact battery life on their users' phone using the data they collected, as reported by TechCrunch:

The new and improved push recommendation feature purportedly only increases battery drain by about 0.7 percent per hour — or, “the equivalent of about a 20-minute game of Angry Birds” over the course of a day.

However, the amount of data collected in this manner is, at the very least, unnecessary. Here's a sample update, lightly modified for legibility and privacy. All coordinates, access point names and MAC addresses were replaced with fictional values. I also transformed the request from x-www-form-urlencoded to json and split a couple of semicolon-separated strings into arrays, for legibility.

Request
POST https://api.foursquare.com/v2/users/updatelocation HTTP/1.1
User-Agent: com.foursquare.android:2013112700:20131127:4.4:Nexus 5:release
Accept-Encoding: gzip
Accept-Language: en-US
Host: api.foursquare.com
Connection: Keep-Alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 1656

{
    "batteryStatus": "unplugged",
    "batteryStrength": 0.59,
    "coarseLL": "40.7250632,-73.9976946",
    "coarseLLAcc": 20,
    "coarseLLTimestamp": 1386052585,
    "csid": 19,
    "hasGpsProvider": true,
    "hasNetworkProvider": true,
    "hasWifi": true,
    "history": [
        "40.7250632,-73.9976946,20.0,1386052041",
        "40.7250632,-73.9976946,29.561,1386052104",
        "40.7250632,-73.9976946,20.0,1386052187",
        "40.7250632,-73.9976946,29.352,1386052247",
        "40.7250632,-73.9976946,20.0,1386052334",
        "40.7250632,-73.9976946,20.0,1386052405",
        "40.7250632,-73.9976946,20.0,1386052465",
        "40.7250632,-73.9976946,29.837,1386052525",
        "40.7250632,-73.9976946,20.0,1386052585"
    ],
    "ll": "40.7250632,-73.9976946",
    "llAcc": 20,
    "oauth_token": (removed),
    "timestamp": 1386052585,
    "token": (removed),
    "trigger": "heartbeat",
    "v": 20131127,
    "wifiScan": [
        "1386052585,Foursquare HQ,00:01:02:03:04:05,2412,-56",
        "1386052585,Another AP,00:01:02:03:04:06,2412,-56",
        "1386052585,Big Apple,00:01:02:03:04:07,2437,-83",
        "1386052585,Other Network,00:01:02:03:04:08,2462,-90",
        "1386052585,dlink,00:01:02:03:04:09,2437,-89",
        "1386052585,linksys,00:01:02:03:04:0a,2412,-89"
    ]
}
Response
HTTP/1.1 200 OK
Date: Tue, 03 Dec 2013 06:36:27 GMT
Server: nginx
Content-Type: application/json; charset=utf-8
Access-Control-Allow-Origin: *
Tracer-Time: 126
X-RateLimit-Limit: 500
X-RateLimit-Remaining: 494
Strict-Transport-Security: max-age=864000
X-ex: fastly_cdn
Accept-Ranges: bytes
Via: 1.1 varnish
Age: 0
X-Served-By: cache-at50-ATL
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1386052587.826462030,VS0,VE144
Vary: Accept-Encoding,User-Agent,Accept-Language
Keep-Alive: timeout=10, max=50
Connection: Keep-Alive
Content-Length: 666

{    
    "meta": {
        "code": 200,
        "requestId": (removed)
    },
    "notifications": [
        {
            "item": {
                "unreadCount": 0
            },
            "type": "notificationTray"
        }
    ],
    "response": {
        "pings": {},
        "primaryDevice": true,
        "signalScan": {
            "iBeacon": false,
            "wifi": false
        },
        "triggers": {
            "nextPing": {
                "geoFence": {                
                    "isNative": false,
                    "lat": 40.7250632,
                    "lng": -73.9976946,
                    "radius": 150
                },
                "minTime": 600
            },
            "notificationTriggered": 0,
            "stopDetect": {
                "accelCeil": 1.0,
                "accelLag": 120.0,
                "accelSigma": 0.005,
                "accelW": 0.0,
                "backgroundTimer": 1800,
                "desiredAcc": 500,
                "fastestInterval": 60,
                "filterRate": 10,
                "highThres": 0.35,
                "kalmanFilter": false,
                "locLag": 60.0,
                "lowThres": 0.2,
                "posSigma": 50.0,
                "sampleRate": 60,
                "speedLag": 150.0,
                "speedW": 1.0,
                "velSigma": 3.0
            }
        }
    }
}

Besides giving us some insights on how their tracking works (triggers object in the response) and how it detects whether the user is on the move (stopDetect), this updates shows just how much data is being shared with Foursquare behind the scenes.

Most of the data is self-explaining, but here's what I could gather by watching a few of these updates:

  • batteryStatus -- one of unplugged, charging or full

  • batteryStrength -- amount of battery charge left (between 0 and 1)

  • coarseLL, coarseLLAcc, coarseLLTimestamp -- current (coarse?) location, accuracy and timestamp

  • history -- set of semicolon-delimited location data points, consisting of timestamp, accuracy and coordinates

  • ll, llAcc -- matched coarseLL, coarseLLAcc in all requests I logged. Probably contains finer (GPS) data when available

  • wifiScan -- set of semicolon-delimited spotted access point data, consisting of timestamp, SSID (network name), BSSID (MAC address) and signal strength

One might wonder if there's a better, less invasive way to deliver the same kind of notifications. Here's a suggestion: have the app download all of the nearby location-specific notifications, then decide locally when to show them. Less creepy and more battery efficient. Win-win.

For now, there is a way to disable this behavior, buried three menus deep under Settings > Account Settings > Privacy:

Privacy settings

Auditing your phone's application traffic yourself

Of course, Foursquare is probably not the only app doing this, which is why you might be motivated to audit your own phone's traffic.

Observing HTTP traffic from a smartphone is simple enough -- fire up your proxy of choice, set it up as a transparent proxy or on the device and watch the logs. That's all there is to it. On the other hand, most interesting traffic occurs over HTTPS for many reasons, including privacy and security. The Foursquare requests described in this article, for example, occur over HTTPS and are not easily intercepted with a common proxy server or packet sniffer.

This section aims to illustrate how to capture HTTP and (most) HTTPS traffic originated by your device, which can be quite useful while auditing third party applications or debugging your own.

Note that not all traffic can be intercepted this way. Fiddler only supports only HTTP, HTTPS and SPDY, which means an application could communicate without showing up in Fiddler by using a different protocol (e.g. a raw or TLS socket). Some applications using certificate pinning might refuse to communicate with this interception in place (e.g. Dropbox and GMail).

The toolset


  • Fiddler -- an excellent HTTP debugger by Eric Lawrence / Telerik.

  • Certificate Maker -- a Fiddler add-on that generates interception certificates compatible with iOS and Android devices. This is only required if you care about HTTPS traffic.

These tools are Windows-only, but there are alternatives for other platforms such as Burp Suite, Charles Proxy and mitmproxy (thanks chubot).

Setting up Fiddler


After installing Fiddler and Certificate Maker, you should be greeted by Fiddler's main window: Fiddler - Main window

HTTPS decryption is off by default in Fiddler. Here are the appropriate settings for this guide (under Tools > Fiddler Options):

Fiddler - HTTPS settings Fiddler - Connection settings

You might need to restart Fiddler after changing these settings. After restarting, click the Capturing icon on the status bar in order to ignore local traffic.

Setting up your device


This guide shows the steps for Android 4.4, but most devices have similar procedures (including iOS devices).

First, open a browser and navigate to your Fiddler instance. The default port for Fiddler is 8888.

Click the FiddlerRoot certificate link, name your certificate and save the changes. This allows Fiddler to behave as a certification authority, generating certificates for intercepted websites on the fly.

Android 4.4 now (rightfully) warns the user when a custom root certificate is installed with a persistent notification. Kudos to Google for that. Since we are the bad guys Google is warning us about, there is no need to be concerned.

Finally, change your proxy settings to have your traffic go through Fiddler.

Capturing traffic


That's it! You should now be able to see requests originated by your phone on Fiddler. Here's an example session:

Fiddler Capture

Try Snapchat if you want to learn how their API works, or any "free flashlight" app if you're feeling brave. Remember to remove all custom settings after exploring.

Comments

comments powered by Disqus
Copyright © 2016 - Thiago Valverde