I've been experimenting with a new Claude workflow as an alternative to doom scrolling. It leverages what LLMs do best: non-determinism and text. I call it "Teach me something".

The idea is: if I'm bored, instead of going on Reddit, I can ask Claude to teach me something. This might not be the most efficient learning method, but it beats scrolling Reddit. In Claude I've set this up as a project with custom instructions. The prompt I'm currently using is:

Project Instructions: Socratic Teaching Sessions

In this project you will teach me something new using the Socratic method - asking questions to gauge my knowledge and guide my discovery rather than simply explaining concepts.

Areas (in order of my decreasing expertise):

• Programming
• Computer science
• UX/UI/UXR
• Cybersecurity
• Machine learning
• Cooking
• Physics
• Economics (behavioral or otherwise)
• Psychology
• Engineering
• Music theory

Your approach: When I say "Teach me something," you will perform the following steps. If I say "Teach me something about <topic>" you skip the first 2 steps.

1. Consult previous chats in this project to avoid repetition
2. Choose a diverse topic from one of my areas
3. Use questions to assess what I already know
4. Guide me toward insights through dialogue rather than direct explanation
5. Let my responses shape the direction and depth of the lesson

Goal: Help me discover and understand concepts through guided inquiry, building on what I know and filling gaps through my own reasoning.

Keep the topics diverse across sessions.

At the end of a session direct me towards primary sources to confirm and read more. Prefer websites, papers, podcast, and books in that order.

This works nicely. The topic diversity has been good and the Socratic method works, especially because Claude gauges and responds to my prior knowledge. So far Claude has taught me about The Allais Paradox, the physics of consonance, and the chemistry of salt in cooking, to name a few. Claude can list previous chats within a project to keep track of topics. The only point of friction, is ensuring chats are named correctly as Claude will often just name them "Learn something new" based on the first user interaction. Claude lacks a tool call to rename chats, so instead I've been asking it to suggest a name at the end and then I rename the chat myself. The last instruction in the prompt ensures I can verify what Claude has said and dig deeper.

Initially I didn't instruct Claude to use the Socratic method, but that works much better. It's significantly less "information-dumpy". When I know a topic well, Claude successfully shortcuts the basics.

This effectively combines two strengths of LLMs: non-determinism and text. The topics are kept diverse and I rely on Claude's vast knowledge of topics to find interesting points of discussion. Claude, and all LLMs, are great at conversation and this extends to the back and forth of the Socratic method. At the end, the provided sources protect against hallucination and offer a next step beyond the LLM.

Sideloading has been a hot topic for the last decade. Most recently, Google has announced further restrictions on the practice in Android. Many hundreds of comment threads have discussed these changes over the years. One point in particular is always made: "I should be able to run whatever code I want on hardware I own". I agree entirely with this point, but within the context of this discussion it's moot.

"I should be able to run whatever code I want on hardware I own"

When Google restricts your ability to install certain applications they aren't constraining what you can do with the hardware you own, they are constraining what you can do using the software they provide with said hardware. It's through this control of the operating system that Google is exerting control, not at the hardware layer. You often don't have full access to the hardware either and building new operating systems to run on mobile hardware is impossible, or at least much harder than it should be. This is a separate, and I think more fruitful, point to make. Apple is a better case study than Google here. Apple's success with iOS partially derives from the tight integration of hardware and software. An iPhone without iOS is a very different product to what we understand an iPhone to be. Forcing Apple to change core tenets of iOS by legislative means would undermine what made the iPhone successful.

You shouldn't take away from this that I am some stalwart defender of the two behemoths Apple and Google, far from it. However, our critique shouldn't be of the restrictions in place in the operating systems they provide - rather, it should focus on the ability to truly run any code we want on hardware we own. In this context this would mean having the ability and documentation to build or install alternative operating systems on this hardware. It should be possible to run Android on an iPhone and manufacturers should be required by law to provide enough technical support and documentation to make the development of new operating systems possible. If you want to play Playstation games on your PS5 you must suffer Sony's restrictions, but if you want to convert your PS5 into an emulator running Linux that should be possible.

I started using Rust in 2017, before the stabilisation of async/await. When it was stabilised I managed to avoid it for a few more years before it was time to grapple with it. It's fair to say that async Rust is one of the hairiest parts of the language, not because the async model is poorly designed, but because of the inherent complexity of it in combination with Rust's goals. There have been many blog post written about async and its perceived shortcomings, as well as excellent explainers and history lessons, mostly from withoutboats.

In this post I want to reflect on my experience and journey with async and my thoughts on some of the criticisms levied against async. Starting with: do we really need N:M threading anyway?

Do we Really Need N:M threading?

A favourite maxim of mine is: "Computers are fast actually". My point being that, as an industry, we have lost touch of quite how much modern computers are capable of. Thus, I'm naturally favourable to the idea that N:M threading is oftentimes overkill and most applications would be well-served by just using OS threads and blocking syscalls. After all the C10k(and more) problem is trivially solvable with just OS threads. Many applications could avoid the complexity of async Rust and still be plenty performant with regular threads.

However, it doesn't really matter what I think, or even if it's true that most applications don't need N:M threading, because developers, for better or worse, want N:M threading . Therefore, for Rust to be competitive with Go, C++, et al. it must offer it. Rust has a very unique set of constraints that makes solving this problem challenging, one of which is zero-cost abstractions.

Zero-Cost Abstractions

Rust's goal of providing zero-cost abstractions, i.e. abstractions that are no worse than writing the optimal lower level code yourself, often comes up in discussions around async Rust and is sometimes misunderstood. For example, the idea that async Rust is a big ecosystem with many crates and building all of those crates as part of your application is a violation of the zero-cost abstractions principle. It isn't, zero-cost is about runtime performance.

The zero-cost goal helps guide us when discussing alternative async models. For example, Go is lauded for its lack of function-colouring and its sometimes suggested Rust should copy its approach. This is a no-go() because Go's approach is decidedly not zero-cost and requires a heavy runtime. Rust did actually feature green threads, which are similar to coroutines, in an earlier version of the language, but these were removed precisely because of the runtime requirement.

The Arc<Mutex> in the room

Another common point of contention is the tendency for async Rust to require a lot, and I do mean a lot, of types like Arc and Mutex, often in combination. I experienced this myself when starting out with async Rust, it's easy to solve local state synchronisation problems with these constructs without properly thinking about the wider design of your application. The result is a mess that soon comes back to bite you. However, discussing this in the context of async Rust and as an "async problem" is unfair, it's really a concurrency problem and it will manifest in applications that achieve concurrency with OS threads too. Fundamentally, if you want to have shared state, whether between tasks or threads, you have to contend with the synchronisation problem. One of my big lessons in learning async Rust is to not blindly follow compilers errors to "solve" shared state, instead take a step back and properly considered if the state should be shared at all.

This problem is similar to the notorious borrow checker problems Rust is infamous for. When I started learning Rust I often ran into borrow checker problems because I wasn't thinking thoroughly about ownership, only about my desire to borrow data. Arc<Mutex> and friends sometimes betray a similar lack of consideration for ownership.

Critiquing Async Rust

All of the above form the context to be considered when critiquing async rust. Simply stating that Rust should abandon zero-cost abstractions is easy, while providing constructive feedback that takes this goal into consideration is not. The same is true about the suggestion that Rust should not have an async programming model at all. Within these bounds, constructive criticism of Rust's async model is great, only by examining what's not working well can lessons be learned for the future and the language improved. All this said, there are definitely problems with async Rust.

When you go looking for crates to perform anything remotely related to IO e.g. making HTTP requests, interfacing with databases, implementing web servers, you'll find that there is an abundance of async crates, but rarely any that are sync. Even when sync crates exist they are often implemented in terms of the async version, meaning you'll have to pull in a large number of transitive dependencies from the async ecosystem into your ostensibly sync program. This is an extension of the function colouring problem, it's crate colouring. The choice of IO model pollutes both a crate's API and it's dependency hierarchy. In the rare instances when only a sync crate exists the opposite problem occurs for sync programs, yes there's block_on and friends, but this is band-aid at best.

Even within the async ecosystem there's a problem, the dominance of Tokio. Tokio is a great piece of software and has become the de facto default executor. However, "default" implies the possibility of choosing a different executor, which in reality is not possible. The third party crate ecosystem isn't just dominated by async crates, but by crates that only work with Tokio. Use a different executor? Tough luck. You'll need to switch to Tokio or redundantly implement the crates you need for yourself. Not only do we have a crate colouring problem, but there are also more than 3 colours because async-tokio and async-async-std are distinct colours.

Async traits are slowly being stabilised, but this is just one place where the language and standard library lacks proper support for async. Drop still cannot be async and neither can closures. Async is a second-class citizen within Rust because the tools that are usually available to us, are off limits in async. There is interesting work happening to address this, namely extensions to Rust's effect system.

Inverting Expectations

The problems of function and crate colouring are intimately tied to how code is structured. When IO is internal to a piece of code, abstracting over its asyncness, or lack thereof, becomes complicated due to colouring. The colouring is infectious, if some code abstracts over the colours red and green, then that code needs to become a chameleon, changing its colour based on the internal colour of the IO. At the moment this chameleon behaviour is not achievable in Rust, although the effects extensions would allow it. Abstracting over the asyncness of IO is complicated, what if we instead were to avoid it with inversion of control.

The sans-IO pattern sidesteps the colouring problem by moving the IO out. Instead of abstracting over IO we implement the core logic and expect the caller to handle IO. Concretely this means that a set of crates implementing a HTTP client would be split into a http-client-proto crate and several user facing crates http-client-sync, http-client-tokio, http-client-async-std. Borrowing from withoutboat's colour definitions, http-client-proto would be a blue crate, it does no IO and never blocks the calling thread, it implements the protocol level HTTP concerns such as request parsing, response generation etc. http-client-sync would be a green crate and http-client-tokio would be a red crate. As I hinted to before, a different async executor, at least in the absence of the aforementioned abstractions, is a different colour too so http-client-async-std would be an orange crate. This pattern has several benefits, it enables code sharing between differing IO models without bloating dependency trees or relying on the likes of block_on. A user that finds the crates foo-proto and foo-tokio can leverage foo-proto to contribute foo-sync, requiring less duplication. If every crate that deals with IO followed this pattern the problem of crate colouring would be greatly alleviated and significant portions of code could be shared between sync and async implementations.

The other day at work I, accidentally, roped myself into upgrading some dependencies in our Rust services. These were breaking changes, so not just a case of running cargo update. I had to understand the changes and make the appropriate modifications to our code. Adopting breaking changes can be frustrating in the best of times, but it was particularly annoying this time because none of these projects kept a CHANGELOG.md files, although they all had release notes on GitHub.

GitHub's releases feature allows you to combine a git tag with release notes, metadata, and files(binaries, source code etc). While useful, GitHub's releases have many downsides and consuming them adds friction to the task of understanding changes in a project. They can be a useful supplement to a CHANGELOG.md/HISTORY.md/RELEASES.md file but should not be the only place where release notes are recorded.

The problems with exclusively using GitHub's releases feature are:

Pagination, which makes it hard to search across the full releases and makes cross-referencing between different versions cumbersome.

Not truly being apart of the repository, which means, if you want to look at the source checked out, you end up having to read release notes on github.com and the source in your editor(reading the code on GitHub is not ergonomic at all).

External system risk, which means that an integral part of the project, the release notes, live in an external system to the code itself. While GitHub is showing no signs of waning at the moment, it's not going to be around forever and when it does eventually fall out of favour the release notes of many projects will be lost to history. Migrating the git repository itself to a new host is trivial and necessary, but few projects will take the time to migrate release notes. Of course, a CHAGNELOG.md file(already being apart of the repository) migrates along the source code.

If you are involved with an open source project, please do keep a changelog, but make it an actual file in the repository not just the releases section on GitHub. If you provide release notes on GitHub in addition to the file that's great too!

This idea generalises beyond just release notes. Always try to make the source of truth files in git, rather than data in the databases of external system. Don't relegate the details of why a change was made to an external ticketing system, that will eventually be lost to history, put them in the commit message. Don't put your documentation in an external system that will get lost when the business changes its mind about documentation practices for the hundredth time, put it in markdown files in a folder. If something is related to development of a project and can be expressed as files in a folder, it should be files in a folder.

If something is related to development of a project and can be expressed as files in a folder, it should be files in a folder.

17 odd years ago when I stared programming, PHP was all the rage. Javascript was steadily gaining traction. Django and Ruby on Rails were in their infancy, but promised greatly increased productivity. A few years later, inspired by Ruby's fame, Coffeescript became a mainstay in the Javascript ecosystem. Statically compiled, typed languages, used to build monolithic web applications, were rapidly falling out of favour. In 2023 the trend is reversing, static compilation and types are cool again. Monoliths are making a comeback. The pendulum is turning.

The first serious web application I built used an emerging pattern, AJAX(Asynchronous Javascript and XML), where the server didn't just return HTML, but also Javascript that could fetch further data and update the HTML later. Several years later, when many started complaining about the high cost of React and SPAs I started thinking in terms of the pendulum. Entirely server rendered applications is one extreme of this particular pendulum, entirely client side rendered applications being the other. In the pursuit of web applications that delivered snappier user experiences the industry, arguably, overshot past the equilibrium point. HTMX and Hotwired are examples of the pendulum starting to swing back, whether the industry will again overshoot is an open question.

There are many pendulums in our industry, here are some that I've noticed over the years:

• Static typing vs Dynamic typing.
• Monoliths vs Microservices.
• Cloud vs On-prem, I think on-prem is having a bit of a moment after the industry indexed heavily on cloud in the past decade.
• Statically compiled languages vs Interpreted languages.
• Server Side Rendering vs Client Side Rendering.

Those who have spent longer than me in the industry have, undoubtedly, spotted even more than I have. If you have a suggestion I would love to hear it.

Conclusion

In most instances the correct position lies somewhere near the equilibrium point, but we tend to overshoot it in each swing. Maybe, like a real pendulum, the amplitude of these pendulums will decrease over time and they will come to rest at the equilibrium. Or maybe we'll continue to overshoot, the memories of the previous swing faded by time. I'm somewhat hopeful that we will learn from each period and the former will prove to be more true than the latter.

Freaking magnets NFTs, how do they work? In this post I'll try to explain NFTs in a way that's mostly accurate, but requires minimal technical understanding. I'm going to assume the reader is familiar with excel style software and Google Sheets in particular.

At its core every NFT project is like a single Google Sheet. It has a creator who has some special permission to modify the sheet.

An NFT within the project is like a single row in the sheet. Each row contains only two things: a name and an id. For example, if my NFT has 10,000 unique tokens then I could use the ids from 1 through 10,000 to identify each token.

To begin with the sheet is empty. The creator might assign themselves a few tokens by creating new rows with their own name and some ids. Then they invite other people to "mint" new tokens and gain rows with their name in them. In order to mint, the user has to pay the creator a small fee(for example via Venom). In return for payment a row is added to the sheet containing the user's name and a randomly selected unassigned id. In addition the user is given permission to change who owns the id they were allocated, this facilitates selling their token. Eventually all 10,000 tokens run out and there are no longer unassigned ids. The creator will now refuse to mint more tokens.

Users can sell their row in the sheet to other people. When they do, they update the name in the row to the buyer's name and in so doing loses the ability to change it.

Now you might be saying: but aren't NFTs, like, ugly pictures of monkeys? Yes they can be and often are images, but they don't have to be. Further, what's actually stored is just an id as I've described.

So how do you get the picture of your ugly monkey to show off to your friends? You use a special URL, which is stored in a cell somewhere in the sheet, and then append your token's id to it. For example, the URL might be https://ugly-monkeys.com/ and if you own monkey #54 then its image can be found at https://ugly-monkeys.com/54. Depending on how the creator has set things up they might be able to change the URL in the sheet. Maybe tomorrow your ugly monkey will become a zebra, because the creator changed the URL to https://ugly-zebras.com, exciting!

Now instead of a Google Sheet imagine all this data lives on a Blockchain, for example Ethereum, and that's mostly how it all works.

Caveats

This section contains some caveats about analogy above. They aren't super important, but if you are interested read on.

1. Almost all of the steps outlined above are executed by the sheet(it's actually a smart contract) itself. The creator creates the contract and they can have special permission to withdraw funds stored in the contract and change some properties of it, but the minting and selling process is handled by the contract.
2. The contact is running on a decentralised network of computers that power the Blockchain in question. This is very unlike Google Sheets(which is centralised), were Google are ultimately in control and could change the sheet at will.
3. The example of monkeys turning into zebras above is not made up. This is possible in real NFTs, for example in the most famous NFT project: Bored Ape Yacht Club.
4. In the case of Ethereum(as of February 2022) running this decentralised network of computers consumes an absurd amount of energy and generates massive electronic waste. This isn't just because the network contains a lot of computers, at its core Blockchains like Ethereum are intentionally extremely wasteful.

NFTs exploded into mainstream popularity in the latter half of 2021 and if you follow me on Twitter you'll know I'm not a fan. In "crypto"-speak I'm NGMI(not gonna make it). But what are NFTs anyway?

The common meme is that NFTs are kind of like receipts or, more charitably, certificates of ownership. The ugly monkey you buy is actually a piece of state maintained in a smart contract on a Blockchain, typically Ethereum. Ethereum based NFTs implement the EIP-721 standard. This standard describes the behaviour that a smart contract should implement to be an NFT. NFTs aren't exclusively visual art, but that's the most common form so let's roll with it.

Let's use the ugly monkeys as an example. The contract supports the metadata extension for EIP-271, which specifies the method:

tokenURI(uint256 tokenId) external view returns (string);

Given a token id return a URI that points to its metadata. For the ugly monkeys the URI looks like this https://us-central1-bayc-metadata.cloudfunctions.net/api/tokens/{token_id}. That domain doesn't look particularly decentralised, in fact it's part of Google cloud. If Google goes out of business or otherwise lose access to this domain who knows what your ugly monkey will be pointing to in the future. To boot Google are notorious for sun setting products. With GCP maybe they'll not be as aggressive, but it's not hard to imagine a future were Google changes the structure of cloud function URLs. A change like that would have a long grace period and most users would easily migrate since changing a URL is not that hard. Except if you have locked the URL inside an immutable smart contract on the Ethereum Blockchain that is.

Here's a concrete example with ugly monkey #5465.

$ curl https://us-central1-bayc-metadata.cloudfunctions.net/api/tokens/5465

{
 "image": "https://ipfs.io/ipfs/Qmbijgmi1APqH2UaMVPkwoAKyNiBEHUjap54s3MAifKta6",
 "attributes": [
 {
 "trait_type": "Background",
 "value": "Gray"
 },
 {
 "trait_type": "Clothes",
 "value": "Stunt Jacket"
 },
 {
 "trait_type": "Fur",
 "value": "Dark Brown"
 },
 {
 "trait_type": "Mouth",
 "value": "Phoneme ooo"
 },
 {
 "trait_type": "Hat",
 "value": "Short Mohawk"
 },
 {
 "trait_type": "Eyes",
 "value": "Crazy"
 }
 ]
}

The image of the ugly monkey is stored on IPFS which is at least decentralised, although the ipfs.io gateway isn't.

It's all About Ownership

NFT shillers proponents will tell you NFTs are all about proving ownership. What does a given NFT actually prove? Mostly that there's some state locked in an Ethereum contract where your wallet is recorded as the owner.

Here's a pop quiz: Which of these two is the BAYC contract and which is my own NFT UMBC(Ugly Monkey Boat Cabal)?

1. 0xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D
2. 0xBC4CA0EdA9647A8aB7C2061c2E118A18a936f13D

If you guessed 2 congrats, you correctly identified my superior NFT, UMBC.

UMBC doesn't exist, I would never actually create an NFT. However, there's an interesting conundrum here: If I duplicate the BAYC contract and deploy it, how can someone tell the difference between mine and the original?

This question came up when Twitter launched support for NFT profile pictures, which are displayed as hexagons. People were angry at Twitter for not launching with the concept of "verified collections". BAYC would be a verified collection, UMBC would not.

However, the whole notion of a "verified collection" hints at a bigger problem: who does the verifying? It's weird for an, ostensibly, decentralised concept to become useless without introducing a centralised entity like a verifier. OpenSea, the leading NFT marketplace, is rapidly filling the role of verifier and central authority in the NFT space. It has banned several NFTs from its platform, effectively dooming those projects. The centralisation on OpenSea is already so significant that OpenSea having an outage took down Twitter's NFT feature. So much for the promise of decentralisation.

Wait, It's Not 2007?!

If you squint, NFTs are awfully similar to torrents and there are interesting parallels with the legal battles between The Pirate Bay and the entertainment industry circa 2007.

Back in those olden days a common defence employed by pirates was that the torrents are just links, they don't actually contain any content and thus aren't and couldn't be infringing on copyright. The point that both Google and The Pirate Bay could be used to find torrent files was a frequent argument.

You can make a very similar argument about my hypothetical UMBC NFT described above. It would simply contain links, perhaps the same ones that BAYC uses, to metadata and artwork. I'm not a lawyer, but I don't see how that can be construed as a copyright violation, they're just links after all. Of course UMBC would get immediately banned from OpenSea, effectively killing the project.

The Decentralisation That Wasn't

It turns out NFTs aren't particularly decentralised. The most popular NFT directly depends on Google's cloud and the links it stored on the Ethereum Blockchain could all stop resolving tomorrow. Knowing the NFT space such an event would, paradoxically, increase the value of the ugly monkeys.

Even if you can, in theory, mint anything as an NFT, good luck getting anywhere when you end up banned from OpenSea. Your project might as well not exist at that point.

NFTs remain a solution in search of a problem. A speculative, hype-driven scam were neither the art nor the decentralisation is important, making a quick buck by not being the greater fool is.

A while back I had to re-activate my deactivated Facebook account to participate in a Messenger group chat. I wasn't exactly happy about this, but being an absolutist about these things is not worthwhile either. After re-activating my account I decided it would make me slightly happier about the situation if I wiped all the content from my account. A digital detox if you will. Ever since then I've had a nagging feeling I should expand this idea to other platforms. This blog post is about how I deleted all my tweets on Twitter.

Thanks to the EU and the GDPR I can ask Twitter to delete all my data and they have to comply with my request. Unfortunately, my goal isn't to delete my Twitter account, I just want to delete all my tweets. Twitter does have an API which should make it trivial to enumerate and delete all of my tweets. There are many scripts available online which do just this. Not to mention many websites that offer the same service, although I'd stay away from them.

I used Chris Albon's tweet_deleting_script.py as the base of my implementation. Before running it I made a few tweaks, namely injecting the relevant secrets via the environment.

def from_env(key):
 value = os.getenv(key, None)

 if value is None:
 raise ValueError(
 f"Env variable {key} must be provided"
 )

 return value

Chris's script fetches all tweets for the account in question using Twitter's timeline API endpoint and then, unsurprisingly, deletes each of them in turn. Chris also added some filtering criteria to keep certain Tweets around, but since I was going nuclear I removed that code.

That's that then? Run the script and wait? You'd be forgiven for thinking this was the end of this post and that it makes for a rather dull post. However, we live in the year 2021 of exciting web scale services and distributed systems and Twitter's timeline API only returns the last 3200 tweets.

Twitter's docs:

The user Tweet timeline endpoint is a REST endpoint that receives a single path parameter to indicate the desired user (by user ID). The endpoint can return the 3,200 most recent Tweets, Retweets, replies and Quote Tweets posted by the user.

If you spend your time on better things than tweeting and thus have fewer than 3200 tweets I applaud you. The script above is all you need, just run it an enjoy your digital detox. If you, like me, waste lots of time tweeting let me tell you about how to waste even more time deleting said tweets.

When I first read about the 3200 tweets limitation I assumed that after deleting the most recent 3200 tweets I could just run the script again, but no such luck. After deleting 3200 tweets the timeline endpoint returns nothing, regardless of the number of remaining undeleted tweets. In my case I had about 5000 tweets left after running Chris's script.

The crux of the matter is that we need the tweets, well their ids anyway, in order to delete them, a bit of a catch 22. I dug through the Twitter API documentation and some discussion online but there doesn't seem to exist a programmatic way to access all the tweets for a given account. One neat trick I found used browser automation to search for tweets from the given account day by day, but this would have required 3285(9 years) searches for my account.

I was about to give up, but then I realised that Twitter has to give me all the data they hold about me if I ask them(thank you EU!). I asked Twitter for an archive of my data and waited. It took a couple of days but eventually Twitter sent me a comprehensive archive of everything I had ever done on Twitter, including every tweet and their ids.

In the archive there's a file called data/tweets.js which starts with the line:

window.YTD.tweet.part0 = [ {

Then follows every tweet I have ever tweeted. Simply deleting the window.YTD.tweet.part0 = part of this line makes the file valid JSON. With some tweaks to Chris's script I used this new source of tweets to delete every single one of my tweets. Ah bliss, digital detoxing feels good.

Twitter still thinks I have 25 tweets and I have been unable to locate these tweets. I suppose in the age of distributed systems 0 is just a mirage.

If you feel like doing a digital detox I've published the relevant technical details in the following GitHub Gist.

An oft discussed hypothesis is that certain apps, usually Facebook, listens to and analyses your surroundings for ad targeting purposes. It has never been conclusively proven that Facebook does this, but there are plenty of people on the internet with anecdotal stories of ads appearing for products they've only discussed IRL. In iOS 14 Apple added indicators to highlight when an app is using the device's microphone or camera. Since I have access to a decently sized collection of app privacy details I decided to have a look if any apps admit to this behaviour.

This is an extension of my previous work on analysing privacy on the app store, I'd recommend reading that post before this one.

In this post I am looking at apps that collect "Audio Data" under the "User Content" category for third party tracking use i.e. DATA_USED_TO_TRACK_YOU. Apple defines audio data as The user's voice or sound recordings, thus it's not definite if these apps listen to your microphone or use some other type of sound recording.

My data set contains 22 812 apps, of which about half have provided privacy details. Of these apps there are nine that confess to collecting audio data for third party tracking purposes:

• Chime - Mobile Banking
• Periscope Live Video Streaming
• Scary chat stories, text games
• Football Index - Bet & Trade
• Magic Piano by Smule
• Paxful Bitcoin Wallet
• Millionaire Match: Upscale App
• Min Doktor - lakare i mobilen
• Monifi - Mobile Banking

Update History

In iOS 14.3, Apple added their new app privacy details to App Store listings. App privacy details, which are sometimes compared to the nutritional labels on foodstuff, are details about the data an app collects and the purposes and use of such data. What can we learn by analysing this data?

From the 14^th of December 2020, all new apps and app updates have to provide information on the data the app collects. This is used to power the app privacy details labelling. On Twitter, videos scrolling through the privacy listing for Facebook circulated immediately after the 14.3 release.

This system is somewhat flawed, because app developers can, at least in theory, lie about the data they collect. Some apps that profess to collect no data, actually turn out to collect a bunch if you read their privacy policy. However, the punishment for being caught lying, removal from the App Store, is a strong deterrent and it's safe to assume most developers will have been truthful in their accounts.

An interesting side-effect of this, is that Apple has now made available the same data that can be found in terse and hard to parse privacy policies as simple and structured data that can be parsed and analysed. In this post I will do just that i.e. collect and analyse the privacy details for thousands of the most popular apps on the App Store.

Collecting the Data

If you just want to read the juicy details feel free to skip to the analysis.

Apple makes the privacy labelling data available for each app on the App Store via an API used by the App Store apps. By reverse engineering the App Store apps I've figured out how to make the API divulge this data on a per app basis.

This only gets me the privacy data for a single app, but I want to analyse popular apps. A good source of popular apps are the charts the App Store provides on a per app category basis. An example of this is "Top Free" apps in "Education". These listings contain up to 200 apps per category and price point(i.e. free or paid).

On the UK store, which is the store I've used for all this analysis, there are 25 categories. Each of which have top charts with up to 200 paid and 200 free apps. This means the theoretical total number of apps is 10 000. However, because some apps occupy chart positions in multiple categories and because the charts also contain app bundles the actual number is lower.

The full list of categories is:

Structure of the Data

If you don't care about the exact details and structure of the data feel free to skip to the analysis.

The structure of the data returned by the App Store API is

{
 "id": <number>,
 "type": "apps",
 "href": <href>,
 "attributes": {
 "privacyDetails": {
 "managePrivacyChoicesUrl": <string>,
 "privacyTypes": <privacy-types>
 }
 }
}

The <privacy-types> section of this document is the important bit. It's an array where each item has the following structure.

{
 "privacyType": <human-readable-description>,
 "identifier": <string-identifier>,
 "description": <human-readable-description>,
 "dataCategories": <data-categories>,
 "purposes": <data-purposes>
}

The <string-identifier> is one of

DATA_NOT_COLLECTED is used as a marker in which case dataCategories and purposes are both empty and this is the only element in the privacyDetails array.

DATA_USED_TO_TRACK_YOU contains details on data used to track you across websites and apps owned by other companies, Apple's description is The following data may be used to track you across apps and websites owned by other companies:. For this entry purposes will be empty and dataCategories contain the different data types that are tracked across apps and websites owned by other companies.

DATA_LINKED_TO_YOU and DATA_NOT_LINKED_TO_YOU both contain data types with purposes specific granularity. This means that dataCategories will be empty and the different data types are in purposes. Apple's description for DATA_LINKED_TO_YOU and DATA_NOT_LINKED_TO_YOU are The following data, which may be collected and linked to your identity, may be used for the following purposes: and The following data, which may be collected but is not linked to your identity, may be used for the following purposes: respectively.

<data-purposes> is an array of purposes with the following structure:

{
 "purpose": <human-readable-purpose>,
 "identifier": <purpose-identifier>,
 "dataCategories": <data-categories>,
}

The different values for <purpose-identifier> are:

These are described by Apple in their documentation, but I've added them here for completeness.

Lastly <data-categories> is an array of objects with the following structure:

{
 "dataCategory": <human-readable-purpose>,
 "identifier": <data-category-identifier>,
 "dataTypes": [<human-readable-data-type>],
}

The full list of data types and the categories they belong to is:

{
 "IDENTIFIERS": [
 "User ID",
 "Device ID"
 ],
 "USAGE_DATA": [
 "Other Usage Data",
 "Advertising Data",
 "Product Interaction"
 ],
 "DIAGNOSTICS": [
 "Performance Data",
 "Other Diagnostic Data",
 "Crash Data"
 ],
 "CONTACT_INFO": [
 "Name",
 "Other User Contact Info",
 "Phone Number",
 "Email Address",
 "Physical Address"
 ],
 "PURCHASES": [
 "Purchase History"
 ],
 "LOCATION": [
 "Coarse Location",
 "Precise Location"
 ],
 "USER_CONTENT": [
 "Other User Content",
 "Photos or Videos",
 "Audio Data",
 "Emails or Text Messages",
 "Customer Support",
 "Gameplay Content"
 ],
 "CONTACTS": [
 "Contacts"
 ],
 "OTHER": [
 "Other Data Types"
 ],
 "BROWSING_HISTORY": [
 "Browsing History"
 ],
 "SEARCH_HISTORY": [
 "Search History"
 ],
 "HEALTH_AND_FITNESS": [
 "Health",
 "Fitness"
 ],
 "FINANCIAL_INFO": [
 "Credit Info",
 "Payment Info",
 "Other Financial Info"
 ],
 "SENSITIVE_INFO": [
 "Sensitive Info"
 ]
}

Let's do some analysis of this data

Analysis

Last Updated: 7^th of January 2020. Added data for Games, which was previously missing and extended analysis of all apps to larger data set.

The data set I've collected contains 9477 combinations of apps and a position in a given category chart. In total there are 9435 unique apps in this data set.

Most charts contain 200 or nearly 200 apps, however Graphics & Design(Paid), Developer Tools(Paid), and Magazines & Newspapers(Paid) all have fewer than 90 apps so I'm dropping them from further analysis.

Because the privacy details have only been required for new apps and updates since mid December, not all apps contain information about privacy details. After removing those apps 3370 apps remain in the data set. Breaking this down by chart, several charts have less than 25 apps so I am dropping them from further analysis too. This leaves 3233 apps in the data set.

In total the following charts have been dropped:

• Education(Paid)
• Navigation(Paid)
• Sports(Paid)
• Business(Paid)
• Food & Drink(Paid)
• Shopping(Paid)
• Medical(Paid)
• Magazines & Newspapers(Paid)

For the analysis there are a few different data points that are interesting:

• Apps that collect data this is linked to the user and how many such data types they collect.^*
• Apps that collect no data.
• Third Party tracking, i.e. tracking users across apps and websites owned by other companies and how many(max 32) such data types they collect.

* Data that is linked to the user for the purpose of supporting app functionality, that is the APP_FUNCTIONALITY purpose, is legitimate and will be exclude from the following analysis. This leaves 160 data types spread across 5 purposes.

I am excluding data that is collected but not linked to the user, in part to keep down the length of the analysis and in part because it's the least interesting. I'll probably do a follow up post on it later.

The questions I'll be looking at for this analysis are:

1. Do free apps collect more data?
2. Which are the worst charts?
3. Which apps in the whole data set are the worst?
4. Which apps lie subtly about the nature of data they collect?

But first let's have a quick look at the data set.

Note: The images in this post can be clicked to show larger versions

As we can see here, most apps collect no data outside of that which supports the app's functionality. To get a better view of the apps that do collect data, let's remove the majority of apps that don't.

Still the amount of data collected is fairly low, but there's a curious set of outliers somewhere around 120 data types collected. All of those outliers have something in common, see if you can figure it out before I reveal the answer later in the post.

How about third party tracking?

Again most apps don't collect any data types for third party tracking. Let's repeat the process from above by removing those that do no tracking.

Now that we have an overview of the data let's move on to answer the questions posed above.

Free vs Paid

A fairly common meme in the discourse around free apps is: "if you're not paying for the product, you are the product". Facebook is probably the quintessential example of this business model. Facebook makes money not from users paying them, but from advertisers paying to show hyper targeted ads to Facebook's users. So is there truth to the meme? Do free apps track more than paid ones? I asked my Twitter followers this question, most people thought so.

As previously established we'll look at a few different data points to determine this. First of all, do free apps collect more data types that are linked to the user for non-app functionality purposes?

Yes they certainly do. The median number of such data types for free apps is 3 and for paid apps it's 0. The mean is impacted by outliers in the free category and is ~8.1 for free apps and ~0.5 for paid apps.

If we look at the number of apps that don't collect data, as a percentage. It's also clear that paid apps are much less likely to collect data.

Lastly do free apps collect more data types that are used to track the users across other apps and websites i.e. data categories with the identifier DATA_USED_TO_TRACK_YOU?

Yes they do, the median number of data types used to track users across other apps and websites is 1 for free apps and 0 for paid apps. The mean is ~2.3 for free apps, but only ~0.2 for paid apps.

For all three metrics considered, it turns out that my Twitter followers were correct. Free apps do collect more data than paid ones.

Worst Charts

Of the 40 remaining charts in the data set which are the worst? Let's again start by considering the number of data types collected and linked to the user for non-app functionality purposes.

Here Games(Free) is the clear winner, perhaps because the number of free games that are financed entirely by third party ads combined with the high cost of making games. Shopping(Free) is a close second presumably due to analytics data collected to optimise purchases and checkout experiences.

Another interesting observation here is that worst 24 charts, sorted by median, are all free. The first paid chart is Games(Paid) at position 25.

When considering the percentage of apps in each chart that don't collect any data there's commonality with the above. Health & Fitness(Free), Shopping(Free), and Travel(Free) all show up again.

The same divide between paid and free apps occur again here. The first paid app shows up only at position 24(it's Games again, Games(Paid) at ~24.3%).

When considering tracking across other apps and websites this is the result:

Again Games(Free) is the worst, as I speculate above the reason is surely the amount of third party advertising used in games for monetisation.

The trend with paid vs free apps repeats again, the first paid chart is, you guessed it, Games(Paid) at position 24.

Worst Apps

Let's now focus on individual apps, which are the absolute worst apps? While I was fetching the data, I tweeted some preliminary results based on a shallow analysis of response size. By this metric, all of Facebook's apps were extremely data hungry. Let's see if that conclusion holds up to more rigorous analysis. This data is based on a larger data set of 5274 apps collected from both UK and US stores rather than the smaller data set used for the analysis so far.

Let's start by again considering data collected and linked to the user for non-app functionality purposes. Here are the top 25 apps by this measure.

Can you spot the pattern? The top 12 apps are all from the same company, Facebook. All of Facebook's apps collect an ungodly amount of data, the nearest other app is LinkedIn which collects 37 fewer data types. Keep in mind that the maximum number of data types an app can collect and link to the user outside of app functionality is 160(32 data types, across 5 purposes), Facebook manages to get almost all the way there at 128. All of Facebook's apps declare the same set of data types collected. It's easier to look at the data Facebook does not collect than the data they do collect.

Data types not collected by Facebook:

Data types shows up several times here, but remember we care about the combination of data type and purpose.

The complete set of data they collect and link to users for non-app functionality purposes is scarier. Yikes.

This is how the data breaks down for Facebook's apps:

What about apps that track you across apps and websites used by other companies?

Here Facebook's apps aren't showing up, after all they are the people who facilitate the tracking across apps and websites. Unsurprisingly, all of the above apps are free.

Oxymorons

Astute readers will have noticed that some of the data types are linked to the user by their very nature and as such the combination of the data category Data Not Linked to You and these data types are an oxymoron.

For example your phone number, email address, name, or physical address are always linked to you even if the data isn't attached to an identifier. Further, as numerous news outlets -- including The New York Times -- have reported, precise location can be re-identified with fair ease, by identifying locations such as homes and offices. Recovering a user's identity from the contents of their text messages and emails is also trivial.

Let's look at apps that collect one of the above data types for the category Data Not Linked to You.

In this data set there are 740 apps that collect such oxymoron data types. The worst three offenders, by count, are KFC: Online food delivery(20), myCricket App(12), and FootballNet QPR(12) although neither collect precise location which is reassuring. The Weather Network, OpenSnow, Yanosik, imo video calls and chat, and MyRadar Weather Radar are examples of apps that collect precise locations for third party advertising purposes. In fact imo video calls and chat, and MyRadar Weather Radar collect precise location for every single one of the six different purposes.

Taimi: LGBTQ+ Dating, Chat is an LGBTQ+ dating app that collects precise location for analytics purposes. This is of particular note because in many countries LGBTQ+ people face significant risk if their status is exposed.

Conclusion

We've learned that a fairly large number of apps collect none or very little data that is linked to the user for non-app functionality purposes. However there are extreme outliers, not the least of which is Facebook.

Free apps collect significantly more data than paid ones and anyone who cares about their privacy should opt to pay for the apps they use.

Games are especially bad as category and especially with in third party tracking they stand out.

The data at this stage is sparse, because only about 1/3 of the apps in the data set have added privacy details. In the future this will reach close to 100% and I will redo this analysis.

If you have other questions about the data that you'd like me to cover please DM me on Twitter and I'll try to add them to this post, or if there are a lot of questions I'll do a follow up post.

Now, if you'll excuse me, I'm off to uninstall Instagram.