Warning: long somewhat related story that is basically humblebragging, but the summary is that bypassing Twitter ratelimits is not very hard.
I didn't feel like playing around with Twitter's annoying certificate pinning so I just uploaded the Twitter APK to Corellium, turned on what they call the "network monitor", opened the Twitter app since it lets you use Twitter without signing in. I clicked around, searched and viewed tweets. Then I looked at the requests in the log and saw it has a similar guest token process to the website but with a few differences. Anyways, if you recreate these requests, with one IP address you can generate a few OAuth tokens with no expiry per day. These tokens are for unauthenticated users so obviously they have no write privileges but that's not what was needed here. So if you have a proxy provider with a large pool of IPs where you can buy like 1GB of bandwidth you can use a very small percent of your bandwidth allowance and get thousands of tokens/secrets easily, all with their own separate rate limits. It doesn't even matter what IP you end up using the tokens on. Then I followed https://docs.google.com/document/d/1xVrPoNutyqTdQ04DXBEZW4ZW... and the fact that /statuses/lookup.json still allows you to return 100 (!) tweets at once to reconstruct something close to what the 50% Twitter firehose would look like. And Twitter doesn't even block datacenter IP addresses! Was going to display the data at https://firehose.lol but the fact that it required a few hundred requests a second made me feel bad so I didn't end up running the program for more than a few minutes at a time and shut it down.
Looking at (a fraction of) the Firehose for a few minutes was interesting, originally I accidentally forgot to not display tweets labelled possibly_sensitive so I saw some pretty salacious material for a few seconds. Lots of Chinese gambling ads even though Twitter is blocked there, dubious investment promoters, accounts with usernames like FirstnameLastname3781264872 who would tweet three random words at each other every couple of seconds, and a handful of funny tweets.
Nice. This isn't nearly as efficient, but a simpler way to bypass the ratelimit is to use archive.md, which is immune to the ratelimit. It's useful if you don't have an account and just want to see a few tweets here and there.
This is the most impressive project in Nim I’ve seen yet. Rewriting any major front-end, complete with working authentication and handling idiosyncrasies of the private API is a herculean task. For context, Twitter would have a team of two dozen or more supporting what this does, effectively. Kudos to the author for accomplishing this feat!
The frontend uses Karax, which is my favorite frontend/SPA library in any language. It is an absolute joy to use, even if it's a bit rough around the edges.
Yeah, and I immediately realized Nitter was back when I saw 100+ unread tweets in my RSS feed.
I just marked them all as read and moved on. I guess this Nitter outage may be the thing that finally pushes me to find the people I follow on other platforms.
this is exactly my experience for the last week. let's see whether I feel the need to update my instance with this patch. for now I'm just like "oh, another twitter link, let me completely ignore it."
I'll either ignore it or wait until somebody else accesses it and copies the text or takes a screenshot. If nobody does, it's because it's not important enough and I probably don't need to know about it.
And the flagship instance of Nitter is unusable more often than not - the fact that it's working today is likely an artifact of people having given up given the week long total outage.
Maybe you don't use it very often? For months before the recent chaos it was very reliable. The fact that it's working today is only because I fixed it, Twitter didn't change anything.
Timelines are usually blank, to the point that I have removed it from the rotation of instances in my browser redirector extension. I assumed this was a result of being rate limited by Twitter.
I will admit that since I did remove it from the rotation, I don't see it that often these days, mostly when others link to it and then I navigate to a timeline.
I’ve had timelines/tweets fail to load, but a refresh fixed it quickly. If this was a consistent problem did you consider opening an issue to see if they could improve the website?
I feel the same way. Don't know if it's because of the algorithm or something else but Twitter these days seems to be 80% bots shilling crypto and/or simping for Musk.
tinfoil hat time: what if that was the point all along? twitter was a vital space for organizing protests - think arab spring, occupy wall street.
one of the richest guys in the world (who's also very anti-union, btw!) buys it up in a time where inequality is getting worse and worse and social fabrics are starting to tear and makes it unusable.
no more space for organizing. one fewer threat to capital.
He owns the site. If this is truly his intention then all he has to do is turn off the main switch. There's really no reason to give him the benefit of doubt and twist the narrative into "Musk is a genius and everything is going according to plan". The simpler theory is likely to be the correct one – he has never run a social network before and has no idea how to stop bleeding users and cash so is desperately throwing ideas at the wall hoping something sticks.
> If this is truly his intention then all he has to do is turn off the main switch
Putting on a secondary tinfoil hat to prevent stray signals from leaking in: the Simple Sabotage Field Manual suggests that you gum-up the works so to slow things down by a lot. You may not want to make the sabotage obvious, because it'd get fixed immediately (e.g. blow up a factory). Back to the topic: turning off the switch would result in an exodus to other platforms.
I too don't believe Elon is that dumb. He could let the site just be. Especially after rate limits, it really smells something. You wouldn't rate limit a social media platform even if you bills are hard to pay, if you want the platform to be around.
He would be really stupid if that is his thinking, because as we just saw these users would just move to a competitor as long as there is one. Twitter doesn't have a monopoly on short conversation-based social media.
Given the dude paid $44B for a site clearly worth significantly less than that, and then promptly ran it into the ground, your "tinfoil hat time" answer honestly seems like the only rational answer. It checks out on more levels than any of his actions have.
Yes, it seems plausible. It was also funded in large part by the Saudis, American banks, and other wealthy individuals who are all strongly incentivized to hinder activist communication networks. Now everyone who used Twitter has to re-form their networks elsewhere. Mission accomplished. https://www.aljazeera.com/news/2022/10/28/saudis-kingdom-hol...
Why is this downvoted? It makes a lot of sense, and it's not some outlandish conspiracy theory (it's very inlandish) as we've seen rich people do similar things before, more than once, and dictators do it very frequently.
I swear, the bots is only increasing ever since Elon bought the site. I've been getting more and more DM from spam accounts, somehow from Japan too (Whaaat?).
Hey there, just wanted to thank you because you also fixed my Twitter Spaces downloader app[0]! After the API changes the default bearer token I was using (same as yours) stopped working, but after changing the same way you all's back to normal :D
Is this really permanent? I'd love to know more about this bearer token.
Because in the other github issue thread it seemed like every time they found a way around Twitter's safeguards, it was shutdown.
It seems like they've literally hard coded a token into the source code. Meaning thousands of nitter-instances, thousands of users, around the world, will use the same token.
I'm curious that an "unofficial" API has been allowed to continue working, however intermittently, at all. I appreciate using Nitter, but something about it doesn't add up to me.
NewPipe has been working for me for years with occasional breaking changes on YouTube's side that required an urgent update. I watch pretty much everything at 2x speed 1080p and have no issues with buffering.
I generally watch things on 720p, and it's been crashing frequently. Setting the resolution to 1080 usually solves the problem, but transfers the problem to my wonky internet connection.
Probably he loudly threatened to fire his workshop elves so they worked through the nights and slept on old mattresses in the office so they could Make Twitter Scale Again and now all is well in Load Balancer land?
With attest, Twitter can tell if there's an actual mobile device hitting Twitter with that bearer token versus today, you can spoof the client and unless they fingerprint the IP address or other subtleties, it's hard to distinguish fake traffic from real traffic.
Awesome, now my light personal-use scraper works again. Didn't even take a restart!
Hope Twitter soon lets go of the temporary login restriction too. Given that this isn't completely blocked without a login, I'd expect that to be not far from now on. From what I've collected, I hope Twitter'd start selling dumps of their public data for a bit of a win/win with AI companies and Twitter itself.
I think this exists: https://developer.twitter.com/en/docs/twitter-api/enterprise... but it is rumored to be absurdly expensive. Unless prices come down to at least the same order of magnitude as scraping I suspect people will be willing to risk the legal uncertainty involved with scraping.
Right now the Squawker fork of Fritter is working fine. You can export from Fritter and import into Squawker (and presumably vice versa if Fritter catches up). It's in the IzzyOnDroid F-Droid repository: https://apt.izzysoft.de/fdroid/index/apk/org.ca.squawker
The latest CI/GitHub Actions builds have a fix for this implemented already! I'll be publishing another beta once I have enough time to QA all the latest changes on the Fritter and Twitter side.
It's disabled on nitter.net to reduce load while this new solution is being tested.
Check the instance list for updated instances, most of them have RSS enabled: https://github.com/zedeus/nitter/wiki/Instances
That number doesn't mean anything though. That's the number of people who looked at the app at least once. Give it a month and we'll see how much traffic it gets per day.
100M users in 5 days doesn't mean anything? I understand waiting to see MAU & stickiness - but having the fastest-adopted platform of all time is something. Instagram could not have hoped for a better launch - all they have to do now is not screw it up like the other guy.
What reason could he possibly have to give them to Nitter, which has at its primary purpose undermining their conversion and engagement attempts?
The old token was the twitter web token. I suspect the new one is one of the mobile clients. Maybe new tweetdeck. Though probably the iOS client token makes the most sense, being the hardest to rotate on a whim with app store review.
The clients currently work logged out (exempting rate limits) and it's not like you can upload a per user copy of the app to play store/app store, so that root of trust needs to start somewhere which is what the nitter team can extract.
Hmmm... I was thinking you could limit based on the unique identifierForVendor[1], but without a way to verify that a given id is legit this would be easily circumvented. An API whereby Apple cryptographically signs a vendor/device-specific ID so you can effectively rate limit without needing any personal info whatsoever would be nice.
A device specific ID would be a fingerprinting mechanism that would conflict with the privacy goals though? It would then have to be resettable, like the ad identifier already is.
They already have a device specific ID, this isn't asking for a new ID. See the link above. It's not considered a fingerprinting mechanism because it's only specific to a device/app-vendor pair, so can't be shared to fingerprint across apps.
The problem is you can't use it for rate limiting because a bad actor could just generate a random ID and use that. That's why an endpoint for validating a given ID was issued for a particular vendor is required for a privacy-preserving anonymous rate limiting implementation.
This does not seem legal. Stealing an access token to bypass access controls is illegal and I suspect these people didn't get permission to just scrape anything they want.
It in illegal in most jurisdictions to access data which you are unauthorized to.
>Either way, you'd surely agree its a noble pursuit?
Considering it hurts Twitter's profitability by not showing ads, hurts Twitter's metrics by not having people sign in or sign up, hurts users who were accidently signed out from signing in to twitter and having a better user experience, and hurts content creators because nitter doesn't allow you to like or retweet posts. I do not see it as a noble pursuit.
More people need to remember that freedom works both ways. Twitter has the freedom to put up walls on its own property, and other people have the freedom to put up ladders on theirs so they can see over the walls. Although law enforcement always seems to end up biased towards the billionaires, it's not an automatic process and they have to go through some contortions to find an excuse, because the law says you and Elon Musk are equal.
There's the CFAA, which is a blatantly unfair law targeting any computer activity billionaires don't like. I hear it hasn't been used in this way for a while, and not many times ever, but if it does get used on you you'll wish you were dead, but some people seem to be okay with low-probability high-impact risks. You might even be found innocent if you convince the judge you are authorized to access public tweets. It's not like you're running an SQL injection.
There's the DMCA's anti-circumvention clause, but that's written by Hollywood billionaires for Hollywood-bought judges to abuse. Elon Musk can have a fun time trying to convince them his platform is equivalent to Disney to get a favourably corrupt judgement.
Other than that, what's stopping you from sending any request you like to access public data? You can say your user agent is Snoopy the Dog, you can say you would prefer to accept MIME type ascii/emojipoo, you can pass the server 1000 IDs at once even though it won't give you that many, and you can tell it you're the Twitter app on Android.
Remember: I am not a lawyer and this is not legal advice.
Saying that if you ignore the law then it is legal is not a useful point.
Even if the tweets are public that doesn't mean you can steal an authorization token to use an API to query for them. If you hacked into so server and downloaded /usr/bin/bash you still accessed data you were not authorized to even though it was a public binary.
Twitter's terms of service makes it clear that you aren't allowed to reverse engineer the android app to take its token and start scraping twitter.
I didn't feel like playing around with Twitter's annoying certificate pinning so I just uploaded the Twitter APK to Corellium, turned on what they call the "network monitor", opened the Twitter app since it lets you use Twitter without signing in. I clicked around, searched and viewed tweets. Then I looked at the requests in the log and saw it has a similar guest token process to the website but with a few differences. Anyways, if you recreate these requests, with one IP address you can generate a few OAuth tokens with no expiry per day. These tokens are for unauthenticated users so obviously they have no write privileges but that's not what was needed here. So if you have a proxy provider with a large pool of IPs where you can buy like 1GB of bandwidth you can use a very small percent of your bandwidth allowance and get thousands of tokens/secrets easily, all with their own separate rate limits. It doesn't even matter what IP you end up using the tokens on. Then I followed https://docs.google.com/document/d/1xVrPoNutyqTdQ04DXBEZW4ZW... and the fact that /statuses/lookup.json still allows you to return 100 (!) tweets at once to reconstruct something close to what the 50% Twitter firehose would look like. And Twitter doesn't even block datacenter IP addresses! Was going to display the data at https://firehose.lol but the fact that it required a few hundred requests a second made me feel bad so I didn't end up running the program for more than a few minutes at a time and shut it down.
Looking at (a fraction of) the Firehose for a few minutes was interesting, originally I accidentally forgot to not display tweets labelled possibly_sensitive so I saw some pretty salacious material for a few seconds. Lots of Chinese gambling ads even though Twitter is blocked there, dubious investment promoters, accounts with usernames like FirstnameLastname3781264872 who would tweet three random words at each other every couple of seconds, and a handful of funny tweets.