Their solution was anything but, and it was truly innovative (at the time) and their success was well deserved.
It was totally low hanging fruit. Try building a search engine today. The barriers to entry are much harder. Finding useful sites and filtering out spam are much more difficult now than when Google started. The signal to noise ratio on the internet has plummeted.
And that is just the difficulty of building an application. Actually gaining marketshare in search is a whole other problem to solve and also very difficult. Even if you manage to build a superior search engine you’ll have an uphill battle convincing people to use it.
This is what it means to have all the low hanging fruit cleaned out. Google built their application at a time when most people weren’t even on the internet and spam was barely getting started as a social problem. Sure, what they had was remarkable and innovative, but only because everything else at the time was so bad. But now? Different story.
You can actually read Page & Brin’s original paper online [1] and implement it yourself. It’s not very difficult. In fact, it was an assignment question in my 3rd year numerical methods class. Unfortunately if you just point the basic algorithm at a crawl (which you can download for free here [2]) you’re going to get useless results. The spammers are optimized to fool this algorithm (Pagerank) so you need to find a way to filter out the spam. And that is a very very deep rabbit hole!
It was totally low hanging fruit. Try building a search engine today. The barriers to entry are much harder. Finding useful sites and filtering out spam are much more difficult now than when Google started. The signal to noise ratio on the internet has plummeted.
This is what everyone thought in 1999. Search engines were a solved problem in a crowded market where there could be little room for new contender
And whatever limitations we had in search were just inherent in how it all worked and we just had to deal with it.
The existence and popularity of Dogpile was an admission that search engines were not solved yet. We just used all of them all at once understanding they were each imperfect.
Google didn’t solve really search, it solved an at the time fairly new and related problem of low quality websites. Arguing about the inadequacy of early search engine in that context is like arguing that email clients in 1985 should have included spam filters.
But that's not where others failed. Instead, it's coming up with the idea, realizing that it's a truly good idea, that was the important thing. (Remember, Yahoo and other(s) didn't want to buy Google and the algorithm, when they had the chance.)
Maybe if you replace "totally low hanging fruit", with "It was undiscovered winner-takes-it-all markets" I'd agree.
In hindsight, everything is obvious.
But initially, discovering that opportunity — looking at how many people (how few, just 2) did, it wasn't easy.
> Maybe if you replace "totally low hanging fruit", with "It was undiscovered winner-takes-it-all markets" I'd agree.
I don't understand the distinction you're trying to make. I think what's being proposed is that the more undiscovered markets get discovered, the fewer undiscovered markets are left, and the more difficult they are to discover. Pagerank was a good idea that worked.
I'm thinking that whenever an undiscovered market gets discovered, this unlocks new undiscovered markets.
And back at the time, before Google Search, many many related markets didn't, couldn't, yet exist. And, back then, I'd think PageRank could count as one of the few and difficult thing to discover.
I think "low hanging fruit" makes it sound too easy :- )
I think "low hanging fruit" makes it sound too easy :- )
Then you missed the point. It doesn't matter whether PageRank was something trivial any grad student could come up with or if it was on the level of Einstein's General Relativity in difficulty. The point is that it was one basic idea.
Today if you want to dethrone Google you need to overcome the enormous amount of engineering that has gone into Google Search and Maps. You can't do that by just "discovering" something as a grad student. It's going to take thousands (or millions) of engineering hours to achieve.
That is what it means for the low-hanging fruit to be gone. It's like the difference between discovering electricity, as we all know took quite a while but was achieved by a small number of scientists and inventors over a period of a couple centuries, and trying to compete against the modern-day electrical distribution network on your own, which is essentially impossible without some kind of Star Trek alien galactic empire level technology.
> Remember, Yahoo and other(s) didn't want to buy Google
The big guys at that time were blinded by their own success, Google wanted users to do their search and leave the site which was completely opposite to what the Yahoo's of that day wanted to do, they wanted users to hang out on their portal.
And now Google is the one blinded by their own success, who doesn't want you to leave their portal, with tactics like embedded search results and AMP. But where are the people who will come and dethrone them? Probably purchased wholesale by Google or Facebook, sequestered safely away collecting a paycheck and inventing no threats.
All outsized successes look like low-hanging fruit with the benefit of hindsight. If you cherry-pick the biggest business success in any 20 year span throughout the industrial/information ages, I’m sure you will find the initial path to success is always gone by the end of that span. That implies it those successes are always low-hanging fruit. If Google strikes you as moreso, I would suggest that’s only because of the novelty and eventual dominance of the web to everyday life which was not a forgone conclusion.
Try building a search engine today. The barriers to entry are much harder.
I’ve often heard the saying how “everything easy has already been invented” and that it’s so much harder today to invent new things.
I think it’s a fallacy. Things were just as hard in the past.
I owned an ISP in 1998 and there were plenty of search engines at the time. Google invented something unique and innovative and they were rewarded tremendously for it. There were literally hundreds of other companies trying to do the same thing but Google was better. If it was low hanging fruit then Hotbot or any of the other major players could’ve done it.
In 1998 there were plenty of search engines and their use was distributed more evenly. The internet was still fairly new and most people weren't online yet. It certainly wasn't an integral part of most people's life. Google came along and improved on the existing search engines and in doing so wiped out a lot of that competition, becoming what they are today.
A new search engine now has to compete with global network effects, orders of magnitudes more data, Android, and a complex web of interconnected functionality.
Even only looking at the advantage that Google Maps brings to the table totally blows past any barriers present in 1998. Sure, you could build a much better search engine but who's going to use it when it can't give them directions?
Then consider that if you look like you'll end up making some headway, you're likely to just get bought and killed off/integrated.
A new search engine now has to compete with global network effects, orders of magnitudes more data, Android, and a complex web of interconnected functionality.
Are you a pessimist in life? You’re focusing on only the negative aspects of starting a search engine today.
24 years after Google was formed we have orders of magnitude cheaper processing, orders of magnitude better AI, the ability to start small with cloud computing and work your way up from there. We have 24 years of search engine research to a large extent publicly available.
I really think all this “things would’ve been so much easier back then” are simply excuses as to why someone can’t do something today.
Hey Richard, great to hear you guys are doing well! A search engine with more personal control over rests is certainly attractive.
My intention wasn't to dismiss the possibility of popular new search engines. Only to highlight that the environment is substantially more complex/challenging than in 1997.
>Google invented something unique and innovative and they were rewarded tremendously for it. There were literally hundreds of other companies trying to do the same thing but Google was better. If it was low hanging fruit then Hotbot or any of the other major players could’ve done it.
I actually think that their entrepreneur spirit and business attitude made them successful. Other guys thought of their search engines as of technical experiments and hobbies, they weren't serious about it. In another words Google cared more for innovating their search engine, making money and then reinvesting it back in R&D and staying ahead of everybody. The same story was with Microsoft and Digital Research; Bill Gates simply cared more business wise and was more fanatical in making money than Gary Kildall.
>Try building a search engine today. The barriers to entry are much harder. Finding useful sites and filtering out spam are much more difficult now than when Google started.
I'm not a programmer but I can write down on a piece of paper an algorithm which is more efficient than Google's in filtering out "spam" websites. I'm from southeast Europe and for my local market there are numerous ecommerce phishing websites that are popping up on a first page of search results. Some even ranked first. I reported them to Google but 6 months after nothing changed.
Power iteration of Google matrix is the concept to look up. They reduced the PageRank problem to a well known linear algebra problem with a lot of efficient libraries.
It was totally low hanging fruit. Try building a search engine today. The barriers to entry are much harder. Finding useful sites and filtering out spam are much more difficult now than when Google started. The signal to noise ratio on the internet has plummeted.
And that is just the difficulty of building an application. Actually gaining marketshare in search is a whole other problem to solve and also very difficult. Even if you manage to build a superior search engine you’ll have an uphill battle convincing people to use it.
This is what it means to have all the low hanging fruit cleaned out. Google built their application at a time when most people weren’t even on the internet and spam was barely getting started as a social problem. Sure, what they had was remarkable and innovative, but only because everything else at the time was so bad. But now? Different story.
You can actually read Page & Brin’s original paper online [1] and implement it yourself. It’s not very difficult. In fact, it was an assignment question in my 3rd year numerical methods class. Unfortunately if you just point the basic algorithm at a crawl (which you can download for free here [2]) you’re going to get useless results. The spammers are optimized to fool this algorithm (Pagerank) so you need to find a way to filter out the spam. And that is a very very deep rabbit hole!
[1] https://research.google/pubs/pub334/
[2] https://commoncrawl.org/