I worked in a company that used elastic search as main db.
It worked, company made alot of money from that project. It was a wrong decision but helped us complete the project very fast. We needed search capability and a db. ES did it both.
Problems that we faced by using elastic search:
High load, high Ram usage : db goes down, more ram needed. Luckily we had ES experts in infra team, helped us alot.(ecommerce company)
To Write and read after, you need to refresh the index or wait a refresh.
More inserts, more index refreshes. Which ES is not designed for, inserts become slow. You need to find a way to insert in bulk.
Api starts, cannot find es alias because of connection issue, creates a new alias(our code did that when it cant find alias, bad idea). Oops whole data on alias is gone.
Most important thing to use ES as main db is to use "keyword" type for every field that you don't text search.
No transaction: if second insert fails you need to delete first insert by hand. Makes code look ugly.
Advantages: you can search, every field is indexed, super fast reads. Fast development. Easy to learn. We never faced data loss, even if db crashed.
Databases and search engines have different engineering priorities, and data integrity is not a top tier priority for search engine developers because a search engine is assumed not to be the primary data store. Search engines are designed to build an index which augments a data store and which can be regenerated when needed.
Anyone in engineering who recommends using a search engine as a primary data store is taking on risk of data loss for their organization that most non-engineering people do not understand.
In one org I worked for, we put the search engine in front of the database for retrieval, but we also made sure that the data was going to Postgres.
> Anyone in engineering who recommends using a search engine as a primary data store is taking on risk of data loss for their organization.
It is true that Elasticsearch was not designed for it, but there is no reason why another "search engine" designed for that purpose couldn't fit that role.
ES should be thought of as a json key value store and search engine. The json key value store is fully consistent and supports read after write semantics, refresh is needed for search api. In some cases it does make sense to treat it as a database provided the key value store semantics is enough.
I used it about 7 years ago. Text search was not that heavily used, but we utilized the keyword filter heavily. It's like having a database where you can throw any query at it and it would return a response in reasonable time, because you are just creating an index on all fields.
agree with comment. We use ES quite extensively as a database with huge documents and touchwood we haven't had any data loss. We take hourly backups and it is simple to restore.
You have to get used to eventual consistency. If you want to read after writing even by id, you have to wait for the indexing to be complete (around 1 second).
You have to design the documents in such a way that you shouldn't need to join the data with anything else. So make sure you have all the data you need for the document inside it. In an SQL db you would normalize the data and then join. Here assume you have only one table and put all the data inside the doc.
But as we evolved and added more and more fields into the document, the document sizes have grown a lot (Megabytes) and hitting limits like (max searchable fields :1000 can be increased but not recommended) search buffer limits 100MB).
My take is that ES is good for exploration and faster development but should switch to SQL as soon the product is successful if you're using it as the main db.
I just watched Tiangong space station with my bare eyes. Beautiful.
Looking forward to watch 14 starlink satellites tomorrow.
Thank you so much for your website.
There are tons of abondened places like this in Turkey and no one cares about them. They are used as barns these days.
Turkish government probably doesn't even know most of these places but treasure hunters loves them.
If you are interested with this stuff check this youtube channels.
This Turkish guy is a treasure hunter which is illegal. There are many treasure hunter youtubers in Turkey but this guy is the best because he is not faking it.
I enjoy it alot, highly recommended.
No English subtitles but you can understand what's going on.
Thought it was amazing that around Göreme there are just miles and miles of abandoned cave houses with zero protection, not even fenced in. You could just wander through them freely.
OverTheWire wargames. I played first 10 levels, it's kinda fun, kinda frustrating but it made me read manuals of 'grep' and 'find' commands on bash. So it made me a better programmer for sure.
https://overthewire.org/wargames/
I love that game too!
Such a great game, that little game even has a final boss!
I sent miniclip a mail in 2012 asking for how to start programming and developing games and they replied! i still have that email in my inbox. At that time i was in highschool too but didn't start programming or anything related until university lol.
My first full time job as a junior dev was to maintain a legacy GWT project which makes tons of $ every year.Setting up the development environment took 1 week. For some odd reason we couldn't build the project module by module. Build was taking at least 2 mins with the bare minimum module count. I quickly become depressed. Waiting the builds and half way into build an error pops-up and you start again. This was 4 months ago, i literally became a deppressed junior dev who hates his job in 1 month. I started applying for job after a month and quit that job after 2 months.
Problems that we faced by using elastic search: High load, high Ram usage : db goes down, more ram needed. Luckily we had ES experts in infra team, helped us alot.(ecommerce company)
To Write and read after, you need to refresh the index or wait a refresh. More inserts, more index refreshes. Which ES is not designed for, inserts become slow. You need to find a way to insert in bulk.
Api starts, cannot find es alias because of connection issue, creates a new alias(our code did that when it cant find alias, bad idea). Oops whole data on alias is gone.
Most important thing to use ES as main db is to use "keyword" type for every field that you don't text search.
No transaction: if second insert fails you need to delete first insert by hand. Makes code look ugly.
Advantages: you can search, every field is indexed, super fast reads. Fast development. Easy to learn. We never faced data loss, even if db crashed.