Twitter’s Identification of Bots
A study on the prevalence of misleading/malicious bots on Twitter
Originally posted on Medium.
by Sravan Bhamidipati and Alex Heitzmann
Disclaimer: This is an independent project. The findings and opinions expressed here are our own, and do not reflect the views of our employers or anyone other than the two of us.
Over the last few years, bots and trolls have broken past one corner of the internet into the mainstream, and have even impacted the highest levels of government. Today we are learning about professional operations where adults and children alike are paid to manipulate or fool the rest of us on social media. For its part Twitter has been taking various measures to protect its users, including a few controversial ones. It is now seeking the help of anyone with ideas.
In September 2017, Twitter reported on the steps it has taken, the arsenal of tools it can deploy, and the steps it intends to take in the future. Around this time we too became curious about this situation. Since most of the events and efforts were concerned with political interference, we decided to do some investigation into the political sphere of Twitter. In October 2017 we collected profile data of 122 million Twitter users who were following at least one of realDonaldTrump and BarackObama, and where available, their most recent tweet (84 million).
In this article we discuss some of our learnings based solely on the metadata of tweets about their sources, show that Twitter may not be fully deploying the tools it mentioned in that September 2017 blog post, and suggest some steps that Twitter could take to mitigate the proliferation of bots.
When people talk about bots, they mean Twitter accounts or clusters of accounts whose tweets are posted automatically using some Twitter app. Twitter apps are programs registered with Twitter that have the ability to post tweets. Every tweet is posted using a Twitter app, and the Twitter API identifies the app for every tweet with a “source” attribute.
Twitter apps can be third-party websites like Instagram, applications that are downloaded from the internet or an app store and installed on a device, or programs that automatically generate tweets (bots). These apps are commonly created for a number of reasons: for fun or learning or research, as full-fledged alternatives to Twitter’s official apps, as widgets for a website’s users to share specific content (or cross-post) on Twitter. In the more unusual and newsworthy cases they may be created with solely a malicious intent, skirting Twitter’s developer terms.
To create a Twitter app, developers start by filling out a simple form on https://apps.twitter.com/app/new. The form is meant to identify the app with a name, a description, and a website. These identifiers are displayed to users when the app asks for permission to post to Twitter on their behalf.
The name and the website of the app used to post tweets can also be found in the “source” attribute of those tweets when retrieved using the Twitter API.
Our data includes 84.29 million tweets posted using 95465 different apps. The top 10 apps account for 90.28% of the tweets (see table below). Examples of other popular apps are: TweetDeck, a Twitter client that got acquired by Twitter; Gleam and Hootsuite, popular among social marketing teams; and fllwrs and UnFollowSpy, popular among users who take their follower counts and “social media clout” seriously.
Twitter has a Developer Policy that explains the dos and don’ts of apps, along with all the legalese one would expect from such documents. A section titled Clearly Identify Your Service asks developers to not peddle spam or mislead users about their identity.
While Twitter has basic measures to prevent users from signing up with an incomplete email address or a weak password, it doesn’t enforce even the most easily verifiable rules of its developer terms to prevent intentional or accidental misuse.
We were able to trivially register a Twitter app with read and write permissions named like “Twitter for iPhone”, with a registration URL of https://twitter.com/download/iphone, and on behalf of Apple Inc. We also created a bot corresponding to it to learn the limits to posting content on Twitter. Posting one tweet at a time, our bot could tweet at a rate of 3.64 tweets per second.
In the sections that follow we will identify a few recognizable patterns of Twitter app usage that we believe are potentially harmful to the Twitter community.
Counting ASCII and approximately ASCII substrings but excluding all other homoglyphs, 1941 apps in our data had ‘bot’ in their name, 67 had ‘Trump’ and ‘bot’ in their name, and 3 had ‘Obama’ and ‘bot’ in their name. Upon manual inspection of 321 apps that had ‘Trump’ but not ‘bot’ and 28 apps that had ‘Obama’ but not ‘bot’ in their name, we found that many but not all of these were also bots.
7122 apps we found were named using the prefix “erased” followed by an optional “_” followed by 3–8 digits. All provided either “http://<app_name>.com” or the app name itself (which is not even a valid URL format) as their URL. 39672 tweets were from such apps. “erased_234897” was the most used app with 5267 tweets, of which 4969 were announcements about being live on Twitcam, a site that no longer exists. In the data we have, the earliest tweet from such an app was posted on June 12, 2009 and the latest one on October 26, 2017, which is to say, this started several years ago, and was still happening during our data collection.
[Revised thought in Dec 2021: I wonder whether "erased" indicates an app that was blocked by Twitter. I can imagine an internal tool for blocking, which as part of its functionality would remove various public identifiers.]
There is nothing intrinsically sinister about the existence of bots. One of our favorite Twitter feeds is of a bot called infinite_scream, which tweets a long form of “AH” every 10 minutes, and replies with a scream to anyone who tweet at it. Nevertheless, the problems with bots that don’t self-identify and can’t be clearly categorized by humans as such are self-evident.
Twitter’s API doesn’t provide a way to retrieve tweets posted by a specific app, or to retrieve any information about apps beyond their URL and name. This makes it difficult to collect data that could be useful to resolve whether or not a certain app is a bot.
An app uses a name or logo that falsely implies relation to another business or person
Twitter’s official apps are often named like “Twitter for X”, where X could be “Android”, “iPhone”, “iPad”, “Websites”, “BlackBerry®”, etc. This appears to be a convention, and the only reason we suspect these to be official apps is because they are among the most popular sources of tweets in the data we collected. On the other hand, we found 173 other apps named in this convention, including 27 variations of “Twitter for Android”, 23 variations of “Twitter for iPhone” and 20 variations of “Twitter for BlackBerry”.
Below are apps that have a name like “Twitter for Android” and use the URL http://twitter.com/download/android or https://twitter.com/download/android, and apps that have a name like “Twitter for iPhone” and use the URL http://twitter.com/download/iphone or https://twitter.com/download/iphone.
Obviously, most of the apps in the above tables are not published by Twitter, Google, or Apple. It is a common practice to register Twitter apps with names that appear identical to existing app names, by replacing certain common ASCII characters in leet-speak or using homoglyph attack generators, or just by adding extra spaces between words. It is also common for Twitter apps to register using official website URLs that the authors are not associated with.
It is possible that many such impersonations have no malicious intent, and a lot of it is for fun or more likely a thoughtless way to skip deciding app names and creating app websites. However, many other such impersonations could be malicious. There is a risk of users being tricked into authorizing untrustworthy apps to do things on their behalf because the authorization screen has fraudulent information. Malicious bots may pretend to be users tweeting from official apps.
Apps that don’t provide the right URL
Twitter directs users to register their apps with a regular URL and to not use shortened URLs, which tend to mask the destination site, but this policy is not enforced either. We programmatically checked the URLs of all of the Twitter apps in our dataset. Below is a sample of non-existent, shortened, incorrect, and/or misleading URLs provided by apps.
Again, there isn’t necessarily anything sinister about such apps. For example, among apps with a URL corresponding to a non-existent domain, the most popular one is Cloudhopper, which stopped existing after Twitter acquired it. Many others were probably apps used by Twitter developers for their one-off projects.
However, it is in general a bad practice to allow verifiably incorrect data into one’s system, and more so for something like a source of user data. The presence of such data could make internal investigations and auditing more difficult for Twitter. Perhaps more importantly, Twitter provides access to its data through its API, and third parties rely on this data to make a range of decisions from academic research to ad campaigns. Allowing misconfigurations of this nature makes effective use of this data more difficult.
Recommendations to Twitter
There are several actions Twitter could take to increase transparency about the source of content, something it has already been working towards. In this section we make a few recommendations under the assumption that Twitter can afford to mildly inconvenience its developers (a tiny fraction of its users). Because Twitter has a large active user base, most of whom are humans largely using official apps and presumably wishing for a safer platform, we think it may be a safe assumption.
Enforce Existing Policies
The only stringent requirement Twitter enforces on developers for registering their first app is a phone number. There are several ways to get virtual phone numbers, and even as Twitter is said to be blacklisting such services, buying a temporary phone number has never been more convenient. And once a developer is registered, they can create any number of apps.
Twitter could easily add more validations to enforce their existing Developer Policies.
Validate app names to disallow potential spoofing. Given a string, checking whether it has non-ASCII characters or odd spacing, and whether it is “very similar” to any of strings that already exist in our dictionary is not difficult. If the rule set for detecting homoglyph attacks seems too complicated, limit the characters that can be used in naming apps.
Validate upon registration of a new app, and periodically thereafter, that the URLs provided are reachable, and that they belong to the developer. (Ad, analytics, and blogging software have supported multiple simple techniques to verify that a webmaster really owns their site.)
Verify the identity of the organization or developers owning popular apps that drive significant traffic. Airbnb and Uber already do this at scale.
These additional validations may inconvenience some developers. To avoid unnecessarily dampening developer enthusiasm Twitter could roll out these policies gradually, starting with apps that are responsible for the highest volumes of API usage.
Introduce Stronger Policies
There are a few other policies that are simple to enforce at scale that Twitter could introduce that predominantly inconvenience bots and not human users.
Reduce Rate Limits
Current rate limits for Twitter accounts are 1000 direct messages and 2400 tweets per day. The stated purpose of these limits is to “alleviate some of the strain on the behind-the-scenes part of Twitter and reduce downtime and error pages”. Twitter does not specify any rate limits for shorter time intervals, and we are unsure whether it has any such limits or whether it is possible to post all 2400 tweets of a day within a second. The bot we created was able to post 3.64 tweets per second, which is about ten times more than what human beings can comprehend, much less compose. Even a single bot could overwhelm an unlucky human user with abuse in a few seconds.
If Twitter aspires to be a more human-centric platform and not merely a sink for generated content, it should rethink and drastically reduce its limits to optimize the experience of human beings. How many tweets can a human being reasonably post in a minute and for how long might they keep it up? Is any value added by allowing bots to post an order of magnitude more frequently than the most prolific human contributors?
Disable Multiple Accounts Per Email
Twitter currently allows creating multiple accounts using variants of an email address that all map to the same inbox (e.g. addresses email@example.com and firstname.lastname@example.org are equivalent in gmail). Disabling this would make a bot creator’s work twice as hard by requiring that they create an email account per bot account. It might not be much, but any trivially implementable deterrents are worth a trial. At the least, Twitter could cap the number of multiple accounts that can be created per email address.
Improve User Badges and Profiles
Some of the policies suggested above might inconvenience app developers and, in rare cases, even humans with legitimate use cases of tweeting 3 times per second using 10 different accounts from 3 different Twitter apps. If twitter believes such policy changes would be too onerous, the twitter community could at least benefit from more transparency about users (accounts).
Every day we see Twitter fights where users call each other bots. Twitter is in a good position to eliminate such speculative attacks by differentiating between humans and bots. Twitter could extend the user badges and profiles to show a count for “Apps”. Just as clicking on any of the existing counts shows us a list of tweets, followed, followers, likes, lists, etc. clicking on “Apps” could show us a list of apps that were used by the user, the number of tweets posted using each one of them, that can be further explored to see the actual tweets. The app that tweets were posted with could also be made easily discoverable. These changes would give Twitter users a better overview of each user’s tweeting habits and allow them to discover new apps in the twitterverse. It could also consider ideas from third-party apps like BotCheck.me, BotOrNot, and BotOMeter.
This project was written in Python, using its built-in sqlite3 library for data storage and python-twitter to retrieve data from Twitter. The code can be reviewed on GitHub. Code that we felt can be reused for other use cases is in the master branch, and code for ad-hoc analysis in the ‘presidential_analysis’ branch. But this wasn’t a pre-planned project and none of the code is organized well. Reviewers are encouraged to point out any bugs, especially the kind that could potentially question our conclusions.
The first data collection round was between 2017–10–07 and 2017–10–14. During this period we retrieved the profiles, and where available the most recent tweets (at most 1 per user), of about 40.1 million followers of realDonaldTrump.
The second round was between 2017–10–14 and 2017–10–31, to retrieve similar data for about 95.4 million followers of BarackObama. For the set of users following both Twitter celebrities, at the end of this round we have up to two tweets — the most recent, and recent but not necessarily the second most recent.
Our understanding is that Twitter disallows distributing the data we collected. To collect similar data, checkout the GitHub repo, from the master branch run
follower_analyzer/follower_analyzer.py -u <username> -d <path_to_db>
and be patient for a few weeks. The roughly 122 million user rows and their corresponding 91 million last statuses we collected consumed about 100 GB of storage. The number of followers will have increased since that time and storage requirements will be correspondingly larger.
In the early stages of this project, we derived some high-level stats of every “column” of the collected data. It is interesting, but we have not followed up any of them for insights. So it only serves as trivia for now. Here is a copy of those stats.