Hey guys, Slava @ Rethink here. The team is really excited to launch Horizon -- it's based on a lot of feedback from users of very different backgrounds, and we think it will make web development dramatically easier.
I've been up for about twenty-four hours, but I'll be around to answer any questions for the rest of the day.
1. Do you have something analogous to https://www.firebase.com/docs/web/api/ondisconnect? Real-time apps often have some manner of presence indicator. Firebase allows you to say "when this client disconnects, set the value of this key to $BLAH". I noticed that Horizon has http://horizon.io/api/horizon/#ondisconnected, but this is just a client-side indication of when you've disconnected.
2. What are Horizon's transaction properties? How about ordering of appends to a collection?
There is currently no "disconnect-and-set" functionality, but it's on the roadmap and will happen soon. It just didn't land into v1.
Since Horizon is backed by RethinkDB, it's transaction properties are the same as that of RethinkDB. Check out this document for the specifics -- http://rethinkdb.com/docs/consistency/.
Thanks for that. Can you clarify disconnect behavior in general? Do subscribed changesets resume if your internet blips?
Also, there doesn't seem to be a callback arg in Collection#store: http://horizon.io/api/collection/#store. This is worrying. How do I know if a write succeeded?
Mike's `react-rethinkdb` project is really cool, but it's based on exposing the entire ReQL language to the browser. We thought this approach wouldn't work well as an industrial-strength product -- ReQL is designed to be accessed from the backend, and exposing a full query language to the browser introduces a number of challenges. Security is really hard, learning curve is steep, there are DoS challenges, the protocol is too complex to write platform-specific extensions, etc.
On the other hand, Horizon exposes a simple API/protocol that's easy to learn and easy to secure. If you need more functionality, you can import Horizon as a Node.js module and start writing backend code that accesses RethinkDB directly. The learning curve is much easier, it provides an upgrade path for sophisticated apps, and feels like a more robust product.
(I don't mean to disparage `react-rethinkdb`, I think the project is awesome, it's just a philosophical difference in approach)
Could you give me a quick comparison of Horizon and Meteor? I've used Meteor in the past (and loved it), and it looks like Horizon might be fun to learn.
It looks to me like Horizon and Meteor are solving problems in different scopes of the application stack.
Horizon looks to be more like the Minimongo/DDP/Accounts parts of Meteor, and doesn't help with any more of the stack like Meteor does (eg: In Horizon its more up to you to wire it into Angular/React/etc). Meteor is more of a platform and Horizon is more of a real-time client DB and account framework.
If I had to choose between the two, Horizon would be good for if you had a strong opinion/technical need for very specific coupling between UI and DB, and Meteor for when you don't or need to get something out very quickly (eg: this weeks internal CRUD app for a few dozen users).
The FAQ repeats some things I've heard about Meteor -- specifically that its architecture doesn't scale.
But Meteor isn't completely prescriptive. There are several pieces that can be swapped out: You can use Angular, React, or Blaze on the client, for instance.
From what I've read, it doesn't look like Horizon is a complete framework, just a database client/server component that can be used with any framework.
We have a short-burst, write-heavy, mostly cpu-bound, work-load, which according to the various financially incentivized critics would completely knee-cap us for our use of Meteor.
Except we:
1) optimized our database operations
2) relegated cpu-bound work to micro-services that could process work from a queue at their own pace
3) Leveraged the ecosystem to writer smarter and more concise code
Know your enemy and know yourself and you need not fear the result of a thousand battles (Sun Tzu). Meteor is an amazingly productive ecosystem for writing real-time applications. No it will not solve every problem you throw at it.
Just to be clear I'm not hating on Horizon. I definitely will try it in the future.
OK, glad to hear that it does work for you. Sounds like you have a robust architecture.
I don't actually know what is supposed to "not scale" about Meteor. I've just read it multiple times. But because of the nature of Meteor (Really easy for anyone to set up! Anyone can create a real time app in ten minutes or less!), it probably attracts a lot of junior developers. It therefore should have occurred to me to translate "It doesn't scale!" to "I don't know how to scale it and my app is slow!!!". :)
There are two major issues that crop up in my experience.
* Node as a runtime is deeply mediocre, in my experience. You can usually solve this by throwing more (usually virtualized) hardware at the problem.
* Meteor's architecture is (at present) tied into MongoDB's oplog tailing. You can scale it up through sharding, through alternate libraries, and DDP itself is not tied to MongoDB at all. IIRC fixing this was a goal for Meteor 2.0.
Both of these can be worked around. If nothing else, the Meteor web side will talk to anything doing DDP, and DDP is not that hard to deal with.
It comes up because Meteor gets slow faster then most other solutions will get slow. It kind of reminds me in Django how a common mis-step for beginners is using an O(n^2) algorithm on the Django ORM. Meteor is the same way if you aren't using proper indices, or are trying do something that SEEMS doable to the uninitiated (like say pushing ~n thousand JSON objects into a MongoDB set), it's going to give you a bad time.
"2) relegated cpu-bound work to micro-services that could process work from a queue at their own pace"
doesn't this defeat the primary selling point of meteor?
2) diverse and highly integrated ecosystem (libs are written just for meteor)
3) productive development environment (es2015 modules on client/server, no fuss build pipline for sass, jade, angular-templates, angular dependency injection)
There are a ton of secondary benefits, but thats the crux of it. Yes we do sacrifice some of the benefits of Meteor as a monolithic (homogenous?) full-stack solution, but I would argue that they are in fact small, and if you dive into Meteor you'll find it to be pretty modular.
Are the transactions optimistic? How does the logic that concurrently applies/rolls-back these transactions over multiple clients work? I'm very interested in this, do you have some internal documentation on it?
Take a look at this comment: https://news.ycombinator.com/item?id=11716145. TL;DR: currently not optimistic but will be soon; it's fairly simple to do because RethinkDB core has built in support for it. Once the feature lands we'll talk about the implementation in detail.
But without optimistic transactions, what sets Horizon apart from just running the transaction-code on the server? If I understand it correctly, the latency will be the same in both cases, i.e., one network round-trip.
Lack of optimistic updates is a temporary state, though. The vision for Horizon definitely includes this functionality, and we'll make sure it lands ASAP.
Edit: While Horizon assigns versions internally, it looks like they are not currently used for catching concurrent client-side modifications by different users. (I previously wrote that they would cause the "losing" write to fail if it was based on an outdated version of the data, but that doesn't seem to be the case yet.)
Congrats on the release! I've only hada little time to play with it so far as a beta user, but was super impressed and can't wait to use it on a real project soon. Cheers!
Hi, Slava, I was a web developer and now I am a mobile developer. Nobody is developing web applications anymore (at least in developing countries). We are all mobile first. How does android or ios developers take advantage of Horizon ? Can you give a few examples? How does horizon make mobile (native) developers life easier ? web development is slowing down and mobile development is on the rise. Mobile developers don't want to deal with backend development. But it seems like web development is your main focus and mobile is a second thought. Please make it the focus of your work. Meteor is useless to us because it is optimized for web client. I hope Horizon won't fall for this trap. Your bigger audience and potential users are android or IOS developers. Please optimize for them and provide lots of examples to show them how their life can be easier. Thanks.
I'm from a developing country too and that is such an exaggeration. Maybe your career has been focused on mobile, but no way there's overall more native development than web.
Web developer here. Some of the most complicated application are being built for web vs mobile. Within the enterprise web applications, I fail to see how mobile is anywhere near web's presence especially for data intensive apps.
Horizon ships with a JavaScript client library that can be used for building web and mobile apps with frameworks like React, Angular, or React Native. We’re also working on specific platform client libraries that support the Horizon protocol. Client libraries for iOS and Android are coming soon.
> Horizon is distributed as a standalone server for browser-based JavaScript apps. When you're ready to write custom backend code, you can load the Horizon modules directly in Node.js
This is key. Firebase seems great for quick prototypes, but I never built against it because it was unclear (to me at least) how to deal with cases where you want to leave the sandbox and add custom backend functionality that they haven't built out for you. Horizon seems ideal -- it gives you the ease of use of a turnkey system starting out, with the flexibility to customize backend logic/validation later when you need to. That, plus the fact that it's put together by the awesome RethinkDB team, makes it very interesting to me!
I've setup background workers that connect to a firebase DB to perform business logic / backendy type stuff. It works rather well and they even have a queue collection specifically for this type of thing. You could even setup a daemon on a VPS / EC2 that would in turn fire up AWS lambda procs (node or w/e) to perform w/ your firebase data.
Regardless, I'm very excited to play around w/ Horizon as well.
For apps that don't necessarily have the realtime requirement, Kinto [1] is another really cool open-source backend solution that focuses on the offline-first scenario. It's built by the Firefox Sync team and used in production at Mozilla.
That said, at first glance there doesn't seem to be anything stopping anyone from building a caching layer on top of Horizon to enable offline-first functionality. Definitely going to be keeping an eye on this as well!
EDIT: It looks like they are in fact considering offline-first support already [2]!
Offline is much harder than one initially think. More specifically, the big issue us that you almost certainly need a solid algorithm to solve conflicts in order for it to work, otherwise it's quite likely you'll update your data in a matter that is unexpected from a user perspective (assuming that two users edit the same data). The only systems i know of which seem to handle this gracefully is Derby/ShareJS (using operational transformation but still lacking offline support) and Amelisa (using CRDTs and in a very early phase development wise). If this is not built in from the ground up, I fear it's going to be very complicated to build in.
Does Horizon also solve "optimistic updates"? If so I'd love to learn more details. For comparison, Meteor keeps a local datastore that updates immediately when data is mutated and then reconciled with the real database.
The current version of Horizon doesn't do optimistic updates, but it's on the roadmap. Check out https://github.com/rethinkdb/horizon/issues/23. This is relatively easy because RethinkDB itself has support for notifying the client via a feed when a particular update has landed (i.e. the RethinkDB client can correlate the write with a feed message). It just didn't make it into v1, but will happen soon.
I'm continuously impressed by Rethink. When I first used it, I ran into a few issues with changefeeds. Since they work in the open, I could quickly look at the Github issues and see that the fixes/features were already slated for future release milestones. To me, that's a huge win.
Slava, Horizon looks cool, and I'm a huge fanboy of RethinkDB. However, in the demo video you write queries client side. How do you protect against users modifying front-end JavaScript and thus the queries?
// ex
this.props.horizon.order('datetime', 'descending').limit(8).watch()
You can't protect from the client modifying the JavaScript code, and that's ok. Horizon ships with a full security model (http://horizon.io/docs/permissions/) that lets you specify exactly what data each user can and cannot access, which queries they can run, etc. We're going to continue expanding the security model to include DoS protection, and any other issues that may come up.
So while the user can change JavaScript as they please, that's ok; if they try to do something they're not supposed to do, the server will just reject the query.
This is the question I always get stuck on with these "front-end only" frameworks. How do you prevent DoS attacks or scrapers downloading your whole database? What is the information security model?
The sibling comment by ubercore has the right link. And your concern is important, since without security, why not just expose your db to the frontend directly?
At a high level, the permission system:
1. Is a whitelist approach (you can't do anything by default)
2. Allows you to specify query shapes that are allowed to be run by the current user. This prevents queries of the wrong shape from ever hitting the database, helping with scalability.
3. Has a fallback to full javascript rules which can depend on the data on disk. A little slower than the query rules, but allows much more flexibility where you need it. And you can combine them with a query rule to do a rough first pass of what is acceptable.
Ultimately, the permissions system is designed to prevent unfettered database access, but it also has an eye towards performance in the case of DOS attacks. Horizon overall attempts to use only indexed queries, low resource usage operations etc, so you don't accidentally invite DOS attacks .
Now, I'm not claiming we're DOS proof or anything, but I just intend to mention we have an eye towards it when we talk about security.
I would think you'd do this server-side, I've had similar issues outside of JavaScript: users play a game and create a custom client that connects to my server, they send custom packets thereby I have to make sure that they: don't access too much data at once (a user asking for 100 different things in 3 seconds is not even normal in certain scenarios) or the data they're requesting is formatted wrong (a string where a integer is expected?) the solution I guess is server-sided. This issue isn't unique only to client-side javascript, it is relevant in networking as well. So long as the back-end is designed intelligently it could all work out.
Out of the box Meteor does publish the to the browser. But I believe if you remove the insecure package you then control what is published to the browser e.g. usually only a subset of the db.
On the Meteor App I'm currently working on, we limit the data the client sees to what they need, so while it makes a scrapers job slightly easier as they can query the DB, they don't get anymore data than any other approach. If data is displayed to the user they can scrape it.
Security is the same as the clients DB writes are rejected if they are modifying fields in ways that are not permitted (eg: changing a trouble ticket from "submitted" to "approved"). This can make some of the security easier to audit as most of the business logic security is in one place rather than scattered about in dozens of API calls.
While the exact technical details for these real-time DB based front ends are different, the basic thinking is the same of never trust the client and its costly to transmit info to the client.
The nice thing about having the DB directly on the client is that it makes CRUD apps much easier to make as you have to build less bespoke framework to get data to/from the client and verifying things conform to business logic. If you are not building a CRUD app you will want to evaluate other methods for a better fit.
I think it's amazing that it's working with real RxJS streams for changes now - given how Redux store now returns an observable thing, Cycle is Rx based and Angular 2.0 uses a lot of Rx.
This guarantees that things will interop nicely with eachother and we'll be able to use Rx all around.
You might want to check out lovli.js (https://github.com/flipace/lovli.js) -- a Horizon+Redux integration. It's really cool and demonstrates how well Horizon can interop with modern web technologies.
Agreed! I had tons of fun building my own CMS on top of RethinkDB (as well as a logging/journal app for personal use). The only reason I ended up with a wordpress/MySQL solution for the CMS was that building the CMS interface is a bitch and migrating from SQL to SQL was a bit easier.
Sorry, but coming from an embedded world the use of Real-time here seems borderline offensive to me. Anyone care to explain what it means in this context? From the dawn of computer science it has been about strict time guarantees - and I highly doubt this is what's being referred to here.
AFAIK, soft-real time has been a well known term since the 80s. It means that the quality of service degrades the longer it takes for a response to be returned, and eventually a response is worthless if enough time passes.
Hey manigandham, Dalan (dalanmiller) from RethinkDB here,
We have a performance benchmark report we did a couple months ago basically locked and loaded to publish with accompanying scripts and data for transparent review of our methodology. We just have been so busy with Horizon, we had put this on the back burner. Expect to see something within the next couple weeks when we can put some serious attention to it to ensure we do everything just right.
In the meantime, you should check out the whitepaper by BigchainDB which uses RethinkDB for the purposes of consistency and automatic change notifications [1].
Christina from RethinkDB here (I work on all things community)... I'll pass the kind words along to our co-founder Mike G. who worked countless hours on the website, plus Annie Ruygt who does the artwork! It's wonderful to hear positive feedback since we do most of our design in-house :D
I'll second deepanchor. It might seem like such a minor detail, but at least for some people, including myself, it does matter. It signifies something.
That's a good report and I've read it before - but still looking for something about the actual performance (latency, throughput, concurrency) that it can handle.
A websocket connection between client and server is great for real-time apps, but for non real-time app, the unnecessary websocket connections create more work for devops -- specifically DNS and load balancer. Does/will Horizon support that clients only use the Collections API to get/set data without websocket (for real-time sync)?
Saw a demo of this at a realtime meetup in Palo Alto and was pretty impressed. Like everything RethinkDB does I anticipate it will be a fantastic one-two punch of sound engineering and smooth user experience.
How does Horizon handle offline network conditions? Are there network + app states (e.g., cold boot, refresh) under which users might not have access to data?
Looks really promising to me, relatively similar to www.deepstream.io . However, will there be additional clients /are they already in development by the community? So far I only found support for vue.js and React. Polymer elements would be pretty dope... ;)
This looks really cool, but I still don’t understand how Rethink plans to cash out their VCs, though—~$12M, according to Angel List. Cloud hosting is trending toward zero and support/consulting only scales with people. I’m not to trying to be (intentionally) critical; I’m genuinely curious and hoping there is some business model innovation going on here too.
Slava @ Rethink here. Just wanted to point out that only a tiny amount of credit for Horizon belongs to me, as most of it has been a product of tireless work of many people (hey Marc, Dalan, Josh, Daniel, Michael, Annie, Christina, Ryan, Mlucy, Chris, Marshall, and all the awesome beta contributors!)
This isn't false humility either -- I've been super busy with some other aspects of RethinkDB over the past few months, so while I did have a hand in designing and shipping Horizon, maybe less than 5% of it was my doing.
How well does this play with Electron (http://electron.atom.io/ ) as a front-end? I didn't find that many references to it (mostly questions about putting them together).
Congrats on the launch! As a long-time user of Meteor I'm always excited to see other approaches to similar issues.
One thing that I always found a bit lacking with Meteor was pagination. The problem in a nutshell is that the client doesn't know how much data is potentially available on the server unless it requests it, making things like showing "page 1 of 12" type result counts tricky (I talk more about it here: https://www.discovermeteor.com/blog/pagination-problems-mete... )
I'd be curious to know how Horizon deals with that problem?
https://github.com/aldeed/meteor-tabular handles this with custom subscriptions, and it stays up to date by continuously polling the collection without a limit. If you want to get more real-time or more scalable, the problem is that if the total-count subscriptions can't fit in Meteor's memory, every subscription would need to re-request its count from Mongo on every incoming oplog entry, no matter whether it ends up mattering.
Sadly, Horizon doesn't seem to support aggregate queries yet, reactive or not, without loading the records with findAll. Theoretically, Rethink could be even more scalable at this than Mongo, since it could use its index to only "wake up" relevant count subscribers. But this is still on the roadmap:
Generally, it seems Horizon is very limited compared to Meteor, but it's a great fresh start, with a much more well-thought-out and scalable architecture, and I'm looking forward to seeing where it goes.
For convenience, this is the section taken from it that addresses this exact question! (I imagine other people may be thinking this)
How is Horizon different from Meteor?
Horizon has philosophical and technical differences with Meteor that will result in vastly different developer experiences.
Horizon is a small layer on top of RethinkDB with a narrow, purpose-built API designed to make it very easy to get started building realtime apps without backend code. Horizon isn’t prescriptive – you can use any front-end framework without any magic/customizations, and once your app outgrows the Horizon API you can use any backend technology (e.g. Node.js, Python, Ruby) and any backend framework (e.g. Rails, Express, Koa, etc.)
By contrast, Meteor is a much more prescriptive framework. Horizon is a reasonably small component that has a very clean separation with the database and the frontend, while Meteor is a much more monolithic experience.
Another major difference is architectural – Meteor uses a LiveQuery component built by tailing MongoDB’s oplog. This approach is fundamentally limited – it’s impossible to do many operations efficiently, and even the basic functionality is extremely difficult to scale.
Horizon is built on RethinkDB, so the LiveQuery functionality is in the database. This allows for much more sophisticated streaming operations, and scalability is dramatically simpler because the database has all the necessary information to allow for a scalable feeds implementation.
Congrats on the launch, this looks like an amazing product that made immediate sense to me. I have a feeling the Ionic community is going to be quite drawn to this over time.
This looks really great! Does Horizon support pluggable transport? I would like to use Ajax paired with EventSource/SSE over HTTP/2 for bidirectional communication instead of Web Sockets.
There is currently no EventSource/SSE support, but this came up a couple of times. It should be pretty easy to add, and is currently on the roadmap (though for now still unscheduled).
How does access control work? For example, if I have a database containing a table "users", and a table "files", and the files table contains a column "owner" (pointing to the users table), then how do I prevent the client-side database engine from accessing the files which do not belong to the user that is currently logged in?
Check out the docs on security and permissions -- http://horizon.io/docs/permissions/. Horizon ships with a full security model that allows you to protect your data and specify exactly what each user can and cannot access and which queries they can run.
Great thinking from the RethinkDB side. One answer I couldn't find on their website is about the offline and optimistic update. It's really tricky to get right. Having a feed of changes is just a small part of having a robust solution.
The big piece remaining that will help is write timestamps in the database, which we're going to try to land in RethinkDB 2.4. If we were controlling a traditional MVC frontend model and had control over the objects themselves, it would be easier to do optimistic updates. Since we're aiming to be compatible with a lot of frontend frameworks where immutability of the model is important, we have to carefully consider how to do it.
That being said, it's super important, so it's near the top of our priorities
Take a look at this comment: https://news.ycombinator.com/item?id=11716145. TL;DR: currently not optimistic but will be soon; it's fairly simple to do because RethinkDB core has built in support for it. Once the feature lands in Horizon we'll talk about the implementation in detail.
This looks great. I wrote my first complicated Firebase app a couple weeks ago, and looking forward to rewriting a version of it with Horizon to compare the two.
I couldn't find it in the docs, but what is the equivalent to Firebase.ServerValue.TIMESTAMP?
It would be cool to have a object oriented database with prototype support, rather then table oriented. I think Rethink is a step in the right direction.
We pushed a hotfix a few minutes after launch that resolved this issue [1] which you may be experiencing. You should try doing `npm update -g` if you're still on `horizon 1.0.0`. You can check with `hz version`.
If this doesn't fix it for you, please open up an issue [2] and I will help you out.
so .. can one write a spreadsheet app that is able to show results reactively in the front-end and will still reflect that same logic in some kind of backend api ?
Kind of being able to have the same logic in my spreadsheet as well as my api ?
I've been up for about twenty-four hours, but I'll be around to answer any questions for the rest of the day.