In this week’s roundup of the latest news in online speech, content moderation and internet regulation, Ben is joined by guest host Mercy Mutemi, lawyer and managing partner of Nzili & Sumbi Advocates. Together, they cover:
People keep accusing me of criticizing Elon Musk because I “hate” him. But I don’t hate him, nor do I criticize him out of any personal feelings at all, beyond thinking that he often is hypocritical in his decision making, and makes decisions that defy common sense and logic. But when he does the right thing, I’m equally happy to call it out positively.
And while I’ve seen some people cheering on the EU’s new investigation of ExTwitter under the DSA (Digital Services Act), I think it’s extremely problematic and hope that Elon fights it. As we’ve explained, the DSA — while more thoughtful and careful in its approach than most US legislation about social media — remains a tool that can be abused for censoring speech.
Supporters of the DSA kept insisting to me that it would never be used that way, while wink-wink-nudge-nudging that if it didn’t magically stop ill-defined bad content online then it had somehow failed.
And thus, it was quite notable when the EU’s unelected technocrat enforcer, Thierry Breton, started threatening ExTwitter and other Silicon Valley companies earlier this year. The most notable thing was that Breton lumped together illegal content (which the sites are required to take down) and “disinformation,” which (in theory!) they’re not required to take down, but are supposed to have some form of best practices for responding to.
Breton lumped the two together, falsely suggesting that websites were required to remove disinformation under the DSA, which would be quite problematic, given that there is no agreed upon definition of disinformation, and often there are extremely conflicting beliefs about what is and what is not disinformation.
The effectiveness of measures taken to combat information manipulation on the platform, notably the effectiveness of X’s so-called ‘Community Notes’ system in the EU and the effectiveness of related policies mitigating risks to civic discourse and electoral processes.
This feels extremely heavy handed and really none of the EU’s business. Community Notes, while not a replacement for a full trust & safety effort, is a really unique and worthwhile experiment (and one that I’d like to see other sites implement as well). How exactly does one judge the “effectiveness” of the system and how is that the EU’s business?
Similarly, this seems really sketchy as well:
A suspected deceptive design of the user interface, notably in relation to checkmarks linked to certain subscription products, the so-called Blue checks.
I mean, yes, Elon fucked up the whole “blue check as a marker of authority” concept by selling them, rather than using it as part of an actual verification system, but again, calling it “deceptive design” seems like a ridiculous statement, and suggests that the EU now feels it’s reasonable to critique product choices by companies.
Even if we think Elon’s choices around this were dumb and wholly counterproductive, that really shouldn’t be for the government to step in and decide.
And, of course, by kicking off this investigation over such silly things, it really undermines what might be legitimate concerns and areas of investigation, making the whole process — and the DSA itself — appear to be less credible.
Still, I can’t help but close this story with a bit of a “told ya so” directed at Elon. Remember, weeks after announcing his intention to purchase Twitter, Elon sat down with Breton and gave a full throated endorsement of the DSA approach. At the time, we warned him that if he really supported free speech, he’d actually be speaking out about the risks for free speech under the DSA (something old Twitter did in pushing back against earlier drafts of it). But instead, he told Breton that he agreed with this approach. And now he’s its first victim.
I hope that he has ExTwitter fight back against this intrusion, as that would help make it clear that the DSA’s rules should not get this deep into the level of tinkering with content on a site or with random features of a site the EU dislikes.
Oh Elon. As we’ve discussed, Elon is infatuated with Community Notes as a sort of crowdsourced alternative to actually funding a trust & safety staff and tooling. And while we actually like Community Notes and think more social media should use similar tools, it’s simply not a full trust & safety replacement.
But, over the past year, we’ve seen that Elon loves to point out when Community Notes supports his priors, and repeatedly claims victory when Community Notes debunks (or even quibbles with) content that Musk doesn’t like. If you look, you can find him cheering on Community Notes time and time again.
Not too long ago, ExTwitter changed the terms of its creator payout system such that creators who regularly get fact-checked via Community Notes will no longer get payouts.
But… how does the man in charge feel about things when he gets fact checked via Community Notes? Well, it appears that his tune quickly changes. While there have been a few times he’s been Community Noted in the past, and he’ll sometimes brush it off with a “yes, even I’m open to having such notes placed on my account,” when it’s a higher profile thing he seems to freak out.
Over the weekend, Tucker Carlson started pushing a very misleading story regarding YouTube sensationalist Gonzalo Lira who made his name as one of those jackass “dating coaches,” (i.e., “pickup artists”) who became a pro-Russia propagandist once the invasion of Ukraine began. Carlson’s version of the story pitched Lira as a “journalist” who was “imprisoned in Ukraine” for “criticizing Zelensky.”
Lira was arrested earlier this year for violations of Ukraine’s criminal code. There are many legitimate questions that can be asked regarding the nature of Ukraine’s laws regarding propaganda and free speech. But, the underlying accusations against Lira seem more focused on how he was revealing the identity and location of both Ukrainian soldiers and western journalists covering the war.
Either way, Musk picked up on Carlson’s story, falsely claimed Lira had been imprisoned for 5 years, and trying to demand answers as to what was happening with him. Community Notes quickly stepped in to first point out that Carlson’s description of Lira’s situation was misleading, and then that Elon’s tweets were also misleading.
After discovering that his own posts were being Community Noted (will he lose access to monetization?), he started claiming that “state actors” were “gaming” Community Notes. And then, hilariously, claimed that this was really a “honey pot” to catch those gaming the system.
The Community Notes folks quickly hit back:
They pointed out that:
Community Notes requires agreement from contributors of differing perspectives, as such is highly resistant to gaming. The entire Community Notes algorithm and data is open source, and can be reviewed by anyone…
Community Notes ftw.
Soon after that, the Community Notes on Elon’s post disappeared. Funny that.
And… soon after that, a different Community Note appeared on Elon’s tweet again pushing back on the idea that Community Notes was easy to game:
So, yes, any such system of crowdsourcing things can be gamed, though ExTwitter’s implementation of Community Notes (a modification of the tool Polis) is done in a way that, at the very least, makes it resistant to such gaming. It’s not impossible to game, but it’s also not easy given the way it’s set up.
But, still, given how often Elon acts like Community Notes is an infallible system that solves most of his trust & safety issues, it’s interesting to note that apparently it’s only “gamed” by “state actors” when its calling out his own false tweets. The rest of the time Community Notes is so accurate that the company can base payment information on it. So, when Community Notes supports Elon’s views, it’s a key part of ExTwitter’s platform strategy. When it goes against Elon’s views, it’s being abused by state actors.
When Twitter first launched what it called “Birdwatch,” I was hopeful that it would turn into a useful alternative approach to helping with trust & safety/content moderation questions, but I noted that there were many open questions, in particular with how it would deal with malicious actors seeking to game the system. When Elon took over Twitter, he really seemed to embrace Birdwatch, though he changed the name to the pointlessly boring “Community Notes.”
I still think the concept is a good one, and think it’s one of Elon’s few good moves. I think other social media sites should experiment with some similar ideas as well.
The problem, though, is that Elon seems to think that Community Notes is an effective replacement for a comprehensive trust & safety program. At the heart of so many of Elon’s decisions in firing the vast majority of the company’s trust & safety staff was that “Community Notes can handle it.”
As we’re in the midst of a series of major crises around the globe, where the flow of information has proven incredibly important, one thing we’re clearly learning is that Community Notes is not up to the task. Just to drive this point home, over the weekend Elon himself posted some fucking nonsense (as he’s prone to do) and many hours later Community Notes pointed out it was hogwash. Elon, as he’s done in the past when he’s been “Noted,” claimed he was happy it happened to himself… before claiming that his post was “obviously a joke meme” and that “there is more than a grain of truth to it.”
So, first of all, there isn’t “more than a grain of truth to it.” The whole thing is simply false. But, more importantly, if you look at the top replies to his “obviously a joke meme,” suggests that Elon’s biggest fans did not, even remotely, think that this was “obviously a joke meme,” but rather took it entirely seriously, cheering him on for “telling the truth.” Here’s just one of the top replies to his original tweet:
Also, it took quite some time for the note to appear on Elon’s account. And, look, content moderation at scale is impossible to do well and all that, but Community Notes seems like the exact wrong approach in situations like this one. Especially at a time when the accounts pushing out the most viewed news these days seem to be made up by a combination of grifters and idiots:
Online we have seen many users of X describe their experience of this crisis as different. Some of that may result from the more ambiguous nature of the larger conflict, especially as the news cycle moves from the unambiguous horror of the initial attack to concerns about Israel’s response. However, our investigation here suggests an additional factor: in Musk’s short tenure as owner of the platform, a new set of news elites has emerged. These elites post frequently, many sharing unvetted content and emotionally charged media. While sharing no single political ideology, many embrace a similar culture of rapid production of unlinked or ambiguously sourced content, embracing a “firehose of media” ethos that places the onus of verification on the end-user. This occurs in an environment that has been shorn of many of the “credibility signals” that served to ground users in the past — checkmarks that indicated notability, fact-checks distributed through Twitter Trends, and Twitter/X-based labeling of deceptive content. Even fundamental affordances of the web — such as simple sourcing through links — have been devalued by the platform, and, perhaps as a result, by the new elites that now direct its users’ attention.
Leaving aside the significant concern of taking away professional, trained trust & safety employees, and replacing them with random (often hand-picked) untrained volunteers, there are serious concerns coming to light about how Community Notes actually works in practice.
Multiple reports have come out lately highlighting the limitations of Community Notes on important breaking news in the midst of various conflicts around the world, where you have malicious actors seeking to deliberately spread misinformation. A report at Wired found that Community Notes is actually making some of the problems worse, rather than better.
On Saturday, the company wrote on its own platform that “notes across the platform are now being seen tens of millions of times per day, generating north of 85 million impressions in the last week.” It added that thousands of new contributors had been enrolled in the system. However, a WIRED investigation found that Community Notes appears to be not functioning as designed, may be vulnerable to coordinated manipulation by outside groups, and lacks transparency about how notes are approved. Sources also claim that it is filled with in-fighting and disinformation, and there appears to be no real oversight from the company itself.
“I understand why they do it, but it doesn’t do anything like what they say it does,” one Community Notes contributor tells WIRED. “It is prone to manipulation, and it is far too slow and cumbersome. It serves no purpose as far as I can see. I think it’s probably making the disinformation worse, to be honest.”
The report isn’t just based on random Community Notes users, but looking more closely at how the program works, and the ability for it to be gamed. Wired found that it wasn’t difficult to set up multiple accounts controlled by one person which all had access to Community Notes, meaning that you could manipulate support for a position with just a small group of users controlling multiple accounts.
It also points to earlier (pre-Elon) research that showed that (then) Birdwatch wasn’t used nearly as much for standard fact checking, but was used in political debates by users who disagreed politically with someone who had tweeted.
Back during the summer, the Poynter Institute had a good analysis of the limitations of Community Notes for dealing with real-time misinformation campaigns during crises. Specifically, the design of the current Community Notes has some, well, questionable assumptions built in. Apparently, it looks over your tweeting history and assigns you to a camp as being either “left” or “right” and then only allows a Community Note to go public if enough of the “left” people and the “right” people agree on a note.
“It has to have ideological consensus,” he said. “That means people on the left and people on the right have to agree that that note must be appended to that tweet.”
Essentially, it requires a “cross-ideological agreement on truth,” and in an increasingly partisan environment, achieving that consensus is almost impossible, he said.
Another complicating factor is the fact that a Twitter algorithm is looking at a user’s past behavior to determine their political leanings, Mahadevan said. Twitter waits until a similar number of people on the political right and left have agreed to attach a public Community Note to a tweet.
While that may work on issues where there isn’t any kind of culture war, it’s completely useless for culture war issues, where plenty of disinformation flows. Indeed, the Poynter report notes that a huge percentage of the highest rated Community Notes inside the Community Notes system are never seen by the public because they don’t have “cross-ideological agreement.”
The problem is that regular Twitter users might never see that note. Sixty percent of the most-rated notes are not public, meaning the Community Notes on “the tweets that most need a Community Note” aren’t public, Mahadevan said.
The setup with “cross-ideological” consensus basically seems almost perfectly designed to make sure that the absolute worst nonsense will never have Community Notes shown publicly.
Meanwhile, a report from NBC News also highlights how even when Community Notes is able to help debunk false information, it often comes way too late.
NBC News focused on two prominent pieces of Israel-Hamas misinformation that have already been debunked: a fake White House news release that was posted to X claiming the Biden administration had granted Israel $8 billion in emergency aid and false reports that St. Porphyrius Orthodox Church in Gaza was destroyed.
Only 8% of 120 posts related to those stories had published community notes, while 26% had unpublished notes from volunteers that had yet to be approved. About two-thirds of the top posts NBC News reviewed had no proposed or published Community Notes on them.
The findings echo what a Community Notes volunteer said was X’s lack of response to efforts to debunk misleading posts.
“All weekend we were furiously vetting, writing, and approving Community Notes on hundreds of posts which were demonstrably fake news,” Kim Picazio, a Community Notes volunteer, wrote on Instagram’s Threads. “It took 2+ days for the backroom to press whatever button to finally make all our warnings publicly viewable. By that time… You know the rest of that sentence.”
And when the Community Notes don’t show up until much later, a ton of nonsense can spread:
A post about the debunked White House news release published by a verified account had nearly 500,000 views and no proposed or appended note Tuesday afternoon.The Community Notes system also showed that a user tried to submit a fact-check Sunday on another post including the same known misinformation but that it had yet to be approved, saying, “Needs more ratings.” The post had accrued 80,000 views since Sunday.
In a search for St. Porphyrius Orthodox Church in Gaza, only five Community Notes had been applied to the top 42 posts echoing the debunked misinformation. Several posts from verified users with no notes repeated the claim and got over 100,000 views, while 13 Community Notes had been proposed on posts of the debunked claims but had not yet been approved for publishing.
During the first 5 days of the conflict, just 438 Community Notes (attached to 309 posts from 223 unique accounts) earned a “HELPFUL” rating and ended up being displayed publicly to users. Although it’s impossible to know what percentage of content about the war this represents, the fact that trending topics related to the conflict have routinely involved hundreds of thousands or even millions of posts suggests that a few hundred posts is just a drop in the bucket. The visible notes were generally attached to popular posts — the 309 posts in question earned a combined total of 2147081 likes, an average of 6948 likes per post. The majority of the posts that earned Community Notes (222 of 309 posts, 71.8%) came from paid X Premium/Twitter Blue subscribers, and the majority of the accounts posting them (147 of 223, 65.9%) are X Premium subscribers, who are potentially earning a share of X’s ad revenue based on the number of times their posts are seen and who therefore have a financial motive to never delete misleading content. (Overall, roughly 7% of posts that received Community Notes were deleted during the period studied, but there’s no reliable way of knowing how many of these posts were related to the Israel/Hamas war.)
Again, I really like the concept of Community Notes. I think it’s a very useful tool — and one example (of many) of trust & safety tools beyond simply “taking down” content. But it needs to be part of a wider strategy, not the only strategy. And, the program can’t be setup with such a huge blindspot for culture war issues.
But, that’s exactly how things currently work, and it’s a shame, in part because I fear it’s going to discourage others from creating their own versions of Community Notes.
