I'm a bit skeptical EU-funding a bunch of professors is the way a search engine will be built.
The primary goal for academics is to publish new findings, while what you need to build a search engine is rock solid CS and information retrieval basics. Academically, it's not very exciting. Most of it was hashed out in the 1980s or earlier.
>I'm a bit skeptical EU-funding a bunch of professors is the way a search engine will be built.
Heh, so, funny story...
>A second grant—the DARPA-NSF grant most closely associated with Google’s origin—was part of a coordinated effort to build a massive digital library using the internet as its backbone. Both grants funded research by two graduate students who were making rapid advances in web-page ranking, as well as tracking (and making sense of) user queries: future Google cofounders Sergey Brin and Larry Page.
>The research by Brin and Page under these grants became the heart of Google: people using search functions to find precisely what they wanted inside a very large data set.
I see this a lot in a project I'm involved with where the majority of contributors are from academia.
The incentive is to mostly push novel things, often with questionable practicability, in order to write a paper about it.
But nobody will implement the 'boring' features needed to make the thing generally useful.
The best remote+paying job that I've ever seen online was from a Darpa project (memex project, about search engines). $180K - $250K+. This was ~5 years ago.
I salute your efforts and endorse your search engine. I also recognize that you know what it takes to build a search engine.
I don't think you have deep familiarity with EU academia.
- The primary goal for academia is to influence society. Publishing is a route to that.
- Being head of the EU search engine project would give high academic status
- There are hundreds of articles which you could publish on this project
- Rock solid CS. Would someone like Knuth count as "rock-solid"? Who is better at CS, the person who can implement quicksort because they practiced leetcode, or the person who invented quicksort?
- information retrieval basics. Again, these basics were probably developed in academia.
The skills you say are basic to this endevour are more prevalent in top-quality professors and post-docs than they are in industry.
On top of these skills, you actually need most of all software engineering and architecture experience. I don't think this is common in academia at all. Not in professors, not in PhD-students. You need practical experience building complex software at a large scale. Across that, you need to implement these CS fundamentals.
This is requires far more CS than you'll find in your usual software development effort, for sure, and many CS professors absolutely fit that bill. However, to the same degree it also demands far more on the software engineering side. People out of academia in general, from every time I've seen them build software, have not been all too impressive on that side of things.
Web search has an incredible demand for being well rounded, beyond anything else I've encountered. CS isn't the hard part bottle-necking everything else, it's just one of the many hard parts.
The project can fail, you are correct, but it does not take anything from any other projects, it is just an other initiative trying to contribute in the space.
Parent commenter own search project Marginalia Search [1] could even benefit from it, or even maybe collaborate with it.
It is not a winner-take-all situation, and we need various open initiative in this space to get out of the current conundrum we are in with Google stronghold on search.
Taxes are taking from funds for personal projects, and universities already get a lot of funding in Europe to do research, maybe they should focus on creating a better environment for students to become researchers or entrepreneurs instead.
Overall I think there are better ways to improve search from an EU perspective by doing what they are supposed to be doing:
- create a fair environment for companies to compete in, e.g., take a look how Google, Apple or Meta's assets are set up to make it harder for competitors, break that up
- improve standards in eduction – it doesn't really make sense for all member countries to think of and maintain a good CS curriculum and they all seem to be pretty bad at it
- make it easier to build something and get funded, and reward creating prosperity, don't tax it to death
Just tested marginalia's random mode btw. Pretty cool, reminds me of the internet when I was a kid
CIA? Other people are saying DARPA and NASA got together to fund the NSF which funded the PhDs of the founders of Google, but even that's a bit too indirect IMO. Where does the CIA fit in?
(Plus: for who might not know, DARPA is US defense research, and heavily influenced by the intelligence services needs. Which is not necessarily bad! Just good to understand where and how Google originated. And wrt DARPA, they funded the creation of the internet itself, for whatever matters.
In Europe, things often go slightly different. The Web is a result of CERN, who are also a project partner of OpenWebSearch.EU. Why? Well, better search can also be beneficial for better science, not just for end users wanting to find their way or buying something.)
Who deliberately did not stay in academia to do it.
More to the point, a successful team building a product like a search engine requires roles that academia doesn't really have.
Who is doing product management?
Who is doing product marketing?
etc
This is all applied engineering at this point, not R&D. How does it at all fit into academia's strong suit?
I think that maybe the point is this is not being tackled as a purely economic endeavour (or if it is, it’s in the “indirect” manner), as such, I suspect roles like “product marketing” are probably unnecessary, at least for now.
Also, tell me you wouldn’t love to work on a large project that wouldn’t be subject to the arbitrary whims and promises of the marketing department.
> the point is this is not being tackled as a purely economic endeavour (or if it is, it’s in the “indirect” manner), as such, I suspect roles like “product marketing” are probably unnecessary, at least for now
Which results in an interesting engine nobody uses. Products that start with the tech and then think of selling it fall on their faces for a reason.
They weren't forced to pivot. Gates explained to they how much money is to be made and they changed their mind. For some time they were very excited how unobtrusive and helpful the ads were. Then they realised there's even more money and the rest is history.
Is Google proves anything, it's that greed is real.
I don't think the number of people or even the size of the budget is wrong. A small team can be incredibly powerful and productive if you have the right people. In fact, I think far more often search engines fail from trying to start too big than too small.
The problem is that you need people who actually know how to architect complex software systems much more than you need revolutionary new algorithms. For that, professors are the wrong people. A professor on the team, sure, that might be helpful. Not half a Manhattan project's worth.
Have no fear; all of the actual work will be done by PhD students straight out of undergrad, and most of the actual leadership will be done by a string of recent PhD grads who need results in 6 months because they'll be full time job marketing for the 6 months after that ;-)
It happens all the time in Europe. Collaboration between public and private companies is pretty much a pipe dream in the EU. Some company that actually works on building search technology would achieve way more than a bunch of professors.
I disagree on the budget though. It is basically pocket change.
Arguably the biggest most unsolved problem in search is how to make a profit (or even break even). This can be approached in two ways: You can either try to find some way of making search more profitable, or you can find a way to make search cheaper. I think the latter is a lot more plausible than the former.
A shoestring budget keeps the costs down by design and by necessity. A large budget virtually ensures the search engine becomes so expensive to operate it will never break even.
Why not offer a paid tier? Seems to work for Kagi. Information elites will soon flock to paid search engines, which won’t be much more expensive than a streaming subscription. I pay for Netflix and am willing to pay for a search engine that offers as good a search service as the video streaming offered by Netflix.
The primary goal for academics is to publish new findings, while what you need to build a search engine is rock solid CS and information retrieval basics. Academically, it's not very exciting. Most of it was hashed out in the 1980s or earlier.