In defense of recommendation

Recommender systems have a terrible public image. many people see algorithmic recommendation as harmful. Allegations include:

  • they create filter bubbles, where clusters of people are shown only stuff that confirms their biases
  • they boost low-quality but engaging content. E.g. false info, click bait, rage bait
  • anything else?

the filter bubble idea is weak (see the Reuters article, maybe that recent study on Twitter, the panel thing with Rasmus Kleis Nielsen). however the opposite thing might be true: social media and/or their recommendation algorithms cause people to see a larger diversity of opinions. however the end result is still the same: the content from the opposing side ends up just ethrenching people in their beliefs. Basically we’re going from “algos cause filter bubbles” to “algos don’t cause enough filter bubbles”.

I’m reminded of a Scott Alexander post about trapped priors. I think the idea is that for people to change their mind, it helps if they see ideas from the other side which aren’t too far in the other direction. it needs to happen gradually. boil the frog basically. so a potential problem with social media and/or algorithms is that it creates this structure where you end up seeing the most extreme/worst takes from the other side, and it drives polarization.

(that actually seems pretty credible to me, for the record–but I’m not sure how much of it is unique to social media or algorithms vs traditional media/society)

see also: age of algorithmic anxiety

I don’t have a strong opinion on whether social media as it exists today has been net positive. I am after all working full time on building alternatives to the current landscape.

But I do strongly believe that recommendation algorithms can be a force for good.

I think a big part of the problem is that recommendation algorithms are understandably associated with Big Tech. I think a lot of the concern isn’t about recommendation algorithms per se, it’s about who’s in charge of the algorithms. people have a sense that they’re being manipulated by large corporations. Big Brother.

As a thought experiment, imagine you’re a programmer. Say you created your very own recommendation algorithm for your own personal use. It downloads news articles from a large variety of sources and suggests ones for you to read. you make it suggest articles that haven’t had many shares on Twitter, so the algorithm helps you to find things that you otherwise might not have seen. When you get a suggestion, you give a rating on how good of a suggestion it was (which doesn’t necessarily correlate with whether or not you agree with the article!) Over time, the algorithm learns to suggest things that have a higher chance of receiving a positive rating, while still recommending articles a variety of sources.

if you had the time to do all that, it seems quite plausible (probable, even!) to me that the algorithm would be beneficial. There’s no conflict of interest; you’ve designed it to meet your own needs. Hopefully this illustrates that recommendation algorithms aren’t inherently bad; it’s just a question of how they’ve been designed and whose interests they serve.

Coming back to the real world in which the vast majority of people are not programmers with infinite free time, there are two questions:

  1. is it possible to alter the social media landscape so that people have confidence that the algorithms they use are prioritizing their own interests?

  2. Is the potential benefit of recommendation large enough that this would be worth the effort? (as opposed to abandoning recommendation altogether)

I think yes! The key is competition. Although we can’t get all the way to the extreme of “everyone is a programmer who wrote their own recommendation algorithm,” we could go a lot further in that direction than we currently are. The social media landscape should consist of lots of smaller services instead of a few huge services. If people are unsatisfied with one service due to algorithmic anxiety (or any other reason), they can switch to a new one. If lots of people have algorithmic anxiety, that creates a market opportunity for services which solve that problem (either by not using algorithms, or by finding ways to increase users’ trust in their algorithms–different services can try different approaches!).

(I guess that’s a response to question #1)

TODO:

  • discuss question #2 (or is it unnecessary? the main point here is that we need more competition, and even–especially–people who don’t like recommendation algorithms can agree with that. With more competition, the question of “can algorithms actually be good” can be left to the market–if they can be, then there will be small services which continue to use them)
  • discuss the “how” of #1 more. How in-depth? That’s the entire focus of all my work currently. Maybe just give an abbreviated version.
  • What exactly is the takeaway of this article? is it just “we need more competition”? that’s disconnected from the current title (“in defense of recommendation”). Do I actually care about convincing people that algorithms can in fact be good (question #2)? I think yes. Who am I trying to convince? people who might build these recommender systems? people who might want to try the recommender system(s) I’m building? (probably the latter I guess)

Other notes:

  • mention The Toxoplasma Of Rage | Slate Star Codex – Regardless of if algorithms are involved or not, information flow always has and always will be governed by complex system beyond any one person’s control, and the “default” system is not necessarily good! Even if you get all your information from chronological RSS feeds, how did the creators of those feeds get their information? etc
1 Like

Great post. I’ve thought quite a bit about algorithms and their place in the web as of late.

The Sample is one of the few algorithms that doesn’t make me feel gross when I use it. For a while I didn’t know why that was until I read another one of your posts where you mentioned that The Sample is more of a jumping off point to human-curation.

Then I realized, it’s not algorithms that are the problem, but rather the application architecture of “algorithm-centricity”. Algorithms are awesome, it’s the orientation of them at the center of everything we use that’s the problem.

The reasons why so many feel that sense of algorithmic anxiety or social cooling is because we know that at the very heart of the services we’re using is an opaque machine that is silently analyzing and directing us at scale. We have no idea how it works, we have no meaningful control over it, and they’re all controlled by a handful of people.

There are 2 things I think are worth exploring in this thought space and I’d be curious to hear your thoughts on them:

  1. Experimenting with apps that are algorithmically-aided as opposed to algorithmically-centric. De-centering the algorithm and allowing for usage of an app with or without it.
  2. Experimenting with a type of standardized-algorithm-protocol for apps that enables a BYOA (bring your own algorithm :stuck_out_tongue_winking_eye:) to give users more agency over the algorithms that they use which, as you already said, is a good thing no matter which way you slice it.

Thanks for making The Sample, and I’m looking forward to following your newsletter. :slight_smile:

All the best,

  • Gabriel
1 Like

Hi Gabriel!

I broadly agree with what you’re saying. I’d frame it in terms of user choice: there should be enough options so that people can have the kind of experience they want; whether that means having an algorithm at the center, or an algorithm on the periphery, or no algorithms. In some cases that choice could be available within a single app, but maybe in other cases you get the choice by picking between different apps. Having algorithmic choice within a single app is probably most important when the content being recommended isn’t very accessible from 3rd party apps.

For example, IMO it would be great if Facebook let 3rd parties supply alternative algorithms since a lot of the content being recommended is private and Facebook doesn’t provide an API for 3rd party apps to e.g. get posts from your friends as long as you authenticate yourself. Twitter on the other hand is mostly recommended public content, and they have a really nice API actually. I already use it in Yakread to provide an alternative ranking algorithm for tweets, and it wouldn’t be hard for others to make apps that provide a strictly chronological feed–people just need to do it. (There is also Tweetdeck already which has a chronological feed.)

(Also it would be amazing if Spotify supported 3rd party algos–in that case, access to content is hindered by the need to license music)

I wrote about potentially doing BYOA in Yakread: Outsourcing the ranking algorithm. Though for Yakread, the content being recommended is even more accessible to 3rd parties than tweets are. With tweets, you do have to go through Twitter which introduces some platform risk (e.g. if they were to neuter their API); whereas Yakread only recommends stuff hosted on other websites anyway.

While BYOA may or may not be strictly necessary (I’m not sure yet), I am fairly confident it would be beneficial. e.g. I’m thinking back to when I started out doing recommendation stuff–it would’ve been great as an entrepreneur to be able to develop my own recommendation algorithm and have it plug into an existing app rather than doing all the additional work of making e.g. a music player app (I started out in music recommendation) just so I can plug in my own algorithm.

Maybe the optimal situation is that there’s at least one app/platform which allows BYOA, so anyone who wants to develop their own algorithm without developing everything else can plug into that (maybe Yakread could be it!). But for anyone else who does want to make their own end-user app, it’s probably less important if they decide to allow BYOA. Perhaps if an ecosystem of 3rd party algorithms develops for e.g. Yakread, then there’d be more incentive for other people who develop apps to support BYOA, so they can benefit from the already-existing 3rd party algorithms without having to develop their own. That’d be pretty great I think!

As for your second question about having a standardized protocol for BYOA–I think it’s probably not necessary to try to define a standard up front, but if the ecosystem thing I described in the previous paragraph actually does emerge, then I bet a protocol would come into play naturally.

1 Like

Thanks for the thoughtful reply!

I love the framing of user-choice because I think user-agency is one of the most important battles to fight as the web continues to develop and iterate.

I’m looking forward to seeing what you do with Yakread!

1 Like

Totally! And thanks for chiming in :slight_smile: .

By the way, I recently discovered that Discord supports forum channels – so I’ve decided to migrate all the topics on this website over there and use Discord exclusively going forward for discussion (I’ve already got a small community over there with more activity than this forum, but until now it wasn’t a great place for more structured discussions like this). I copied this discussion over here. Unfortunately you have to be signed in to Discord to see it. There’s an invite link if you’d like to join, though I’ll also be summarizing new posts/discussions from Discord in the weekly newsletter. (And at some point I’d like to write a Discord bot that publishes all the forum discussions to a public website, but probably won’t happen for a while.)