Does anyone else think that all this head scratching over what is the most trivial of problems is hilarious? Twitter is just another naming service. Instead of addressing someone as foo@foo.com, they're now just @foo. All that's needed to make twitter distributed is a new protocol for name resolution. It could be as simple as GET http://distribtwitter.com/whois?foo returning <name>http://www.foo.com/mytwitterfeed</name>. You could write it in a day as a wordpress plugin.
Then other people could come along and write recursive caching distribtwitter name servers so that they can still twitter at people they've never twittered before when distribtwitter.com is down.
The only conclusion I can draw from this is that web 2.0 rots the mind and turns trivial problems in to seemingly impossible problems.
Elegant though it is, I don't think that idea will work; people who I don't "follow" can "message" me in Twitter by including my name in a posting. That doesn't work if I explicitly have to be polling their server to see anything they say.
No, people who you don't follow can ping you saying they've sent you a message. Your software can simply drop them on the floor if you don't care to receive messages from people who you aren't following (AKA people who aren't on your whitelist.)
The open source part is fine, its the distributed part that would never work. I don't think people realize the SMS infrastructure/expenditure needed to pull off something like Twitter.
Their 5 digit short code costs (at least) $1k/month in the US. Twitter has these in multiple countries. Each received message costs approx. $.03 and each sent message costs approx. $.05. Added up over 30mil. messages per month (avg. of $.04 per msg.) = $1.2mil. Twitter is almost certainly getting a better rate, but their monthly messaging expenditure is certainly multiples of $100k.
For those of you saying, 'Oh, but what about sending TXT msgs via email. I do that on my homebrew project, and its great!' Sending 30mil msgs (and growing) WILL NOT work via email, the carriers will NEVER let you do that.
I could go on about the technical reasons Twitter cant be distributed and FREE, but in any substantial mobile play you have to deal with the carriers, their infrastructure, and their fees. It is a totally different scenario than a website that has per page serving costs approaching 0.
If you get enough messages on your shortcode, the carriers start to pay you (since they make so much per a message). This doesn't mean an OOS solution will be able to pull this off, but I just wanted to note that twitter need not be stuck with that bill.
No, the carriers don't start to pay you, ever. The only scenario that leads to the carriers paying you are if you have a Premium (pay per message) billing agreement. Twitter does not.
Twitter has taken a significant amount of funding. They have not really expanded their team much, they havent added many new features. They havent been advertising.
I guarantee all of the money is going towards messaging costs and infrastructure. Two things that a distributed, FREE Twitter would have a very hard time accounting for. [But hey, Im all for it. I have many years in the text messaging space and would totally be into helping however i could. I just think that the solution would be far trickier than people might at first glance suspect.]
I do not understand how someone can look at a primitive 160-character web SMS system and suggest that it was open APIs that made them vulnerable to competitors.
The network effect is Twitter's barrier to entry. Everyone goes where everyone else is. If you can easily move the network elsewhere via the api, the barrier is gone.
Yes, if you manage to convert a critical mass of Twitter API users into Twitter-shadow API users. A similar "technique" converts all AIM users into AIM-shadow users.
Fairly obvious, I guess. It's always surprised me as well why no one has made Twitter into a protocol of sorts with a web app that you can host yourself instead of a centralized service.
(the reason I haven't done that myself is that I have absolutely zero interest in Twittering or Tweets or whatever that stuff is called)
"This means the open source community will almost certainly be more clever about the solution that one tiny closed source company because there will be lots of heads focused on an easy to understand problem."
To be honest, I have to completely disagree. Lots of heads is not a solution to a pure computer science problem. Most companies are formed around a specific solution to a computer science problem. They then make everything proprietary and profit.
That's like saying, only smart computer scientists exist in the open source world. Not true. In fact, you simply cannot claim that there is a strong trend in one or the other. Many smart brains in open source... Many smart ones in industry as well...
Twitter isn't a "computer science problem". It's a nice GUI for subscribing to RSS feeds; RSS feeds that can only contain 140-character posts. Oh, and it can email new entries to your phone.
actually the scaling is absolutely an interesting problem. Its the reason they keep going down. because messages are not point to point it is a much harder problem than instant messaging. I have spent a fair amount of time considering the issue in detail because a friend of mine developed a caching system targeted at microblogging. It is non-trivial, and cannot be done with RSS feeds.
If you think it's a hard problem, you aren't thinking about it correctly.
When you say "@foo Hi, foo!" You're sending a message to some person named foo. That's a push. If you start thinking of it as a pull, then you get in to the sort of trouble you're mired in.
In a distributed system the process goes like this:
1) Sender queries name server: WHOIS foo?
2) Nameserver responds foo is http://www.foo.com/mytwitterfeed.
3) Sender pings recipient http//www.foo.com/mytwitterfeed?ping=messageid&origin=bar
4) Recipient queries name server: WHOIS bar?
5) Name server responds http://bar.com/mytwitterfeed
6) Recipient requests message text, http://bar.com/mytwitterfeed?id=messageid and verifies it actually is addressed to him.
From then on, you have to worry about spam but that's a solved problem (killfiles, Bayesian filtering, etc.) No caching is involved. Recipients store messages addressed to them, just like any other messaging system. This isn't at all a hard problem.
What's your point? You might as well say HTTP is a method that allows strangers to get files from my server! While technically true, it doesn't say much.
Yes, people can send you false pings, which is why you need to go back through the name server and do a reverse lookup, and once you fetch the message you need to parse it and make sure it within bounds (less than 160 characters or whatever) and addressed to you (has @foo in the message text.) and is not spam (the name isn't in your killfile, is in your whitelist, akismet says it isn't spam, etc.) Rate limiting helps too, real people aren't going to send you 100 messages a second.
IMO: They are dealing with a tiny amount of bandwidth say there are 30million 1kb text messages a day that's 900 gb / month which is well easy for a single core CPU to handle.
Personally I would go with:
User Adam sends message to Bob:
1) Adam sends message to Twitter service. "Bob>Hi."
2) Twitter check to see if Bob knows Adam.
3) Twitter sends "Adam>Hi." to Bob.
(With optimal storage so Bob and Adam can see their old conversations)
Your system let's random spammers people find out people's address which IMO is bad.
PS: Or Adam could send a message to "All>I like this soup." and Twitter then sends the message to everyone that cares about Adam including Bob.
Edit: It looks like Twitter is sending around 2million Tweets a day.
You're thinking too qualitatively; imagine the spikes they get... Also most apps are I/O restricted, not CPU...
Also, all these proposed messaging architectures are somewhat flawed. Twitter isn't really "sending" messages to other people. It's more like a person's message history is bound to their account, and appropriate privileges are applied. Then, when people try to "read," privileges are obeyed and information is produced...
Am I the only one who believes the solution to Twitter does not involve a massive distributed system? Twitter is inherently centralized... people don't see that.
Ok I can't edit the old post but they are basically just reinventing mailing lists. Anyway, poling is fine if it's initiated by a user action or it's a vary limited number of systems but if you have a few million users you can't uses it per user. As to reading old log's that's a trivially parallizable problem see database replication for some hints. So you separate the active messaging that automatically adds new Tweets to a user’s cell phone from the way users read old messages with the webpage and your home free.
PS: Text is cheep EX: Slashdot is running off of ~4 computers. And bandwidth is not a scaling issue until you need a single system to handle more than ~1GB of bandwidth per second. (Saturating an OC-3 line costs a lot of money but you don't need to change your architecture to do so.)
While I confess that I am an ubernerd (in addition to other fine qualities such as being a well-known chocoholic), I see plenty of people on twitter who are not techie at all.
I follow a few quilters, for example; one of them has a blog with neat how-to videos. And for some reason a lot of real-estate people from Phoenix are on twitter (they follow me, I'm not too interested in them).
I realize that they can be invisible to techies. Like LinkedIn: you only see the people on twitter who you follow personally, so there's a sense that everyone is like yourself. But that's not the case.
Let's do it. I just registered the twitshadow domain and I'm willing to dedicate some of my company's resources to the project. I'm thinking about what would be the best language to do it in. My first choice would be Python.
It's way too early for choice of language. But if I had to chime in, I'd say Erlang would be a fine choice, especially that we have ejabberd to build from.
This was actually a really interesting posit. I think that it's sort of like instant messaging. Someone is going to create the Jabber of Twitter technology but I think there will still be room for the Twitters and Jaikus to run the game.
Then other people could come along and write recursive caching distribtwitter name servers so that they can still twitter at people they've never twittered before when distribtwitter.com is down.
The only conclusion I can draw from this is that web 2.0 rots the mind and turns trivial problems in to seemingly impossible problems.