The core of Meteor has always been its advanced yet easy to use data system, which makes it very easy to efficiently publish reactive data. Looking around, it’s clear to us that many developers across the wider software community – for example, apps that use a combination of Angular and Spring, or React and Rails – have similar needs. We’re eager to bring that technology to a broader array of apps, and we’re also excited to upgrade Meteor with better control over reactivity, support for more data sources and clients, and enhanced productivity for developers.
We have listened to all of the feedback and ideas from the Meteor community’s experience with the current system, talked to production users of Meteor and other platforms, thought about our own experience building apps like Meteor Galaxy, and put together all we have learned in the last 4 years about building modern connected-client apps.
Common problems with data loading
Multiple data sources - SQL, REST, and MongoDB
Meteor currently only supports MongoDB. First-class support for other databases and backends has been one of the top most-requested features, and there are two main problems to be solved. The first is that your UI should be able to work independently of how the backend data is stored, which means you need a unified query language that can work to fetch data from many different sources. The second is that the number of data sources will always be larger than the number of maintained libraries, so it needs to be easy to write your own connector to any database or web API without needing a PhD in systems engineering.
Associations between objects
Most applications have to deal with combining associated objects. In Meteor’s DDP, you have to write complicated publications that publish all of the related data at once. In REST, you have to make multiple roundtrips to the server since each endpoint only returns one type of item, or you need to break out of the REST model and write custom endpoints. Wouldn’t it be great if your data system supported these associations as a first-class feature, along with other things that server-side frameworks take for granted, like pagination and aggregations?
Many apps today have collaborative UI features that need a live view of data as it changes on the server. Four years ago, Meteor made the audacious choice to have live-updating as the default, and one of the hallmarks of a Meteor app is an engaging, collaborative experience. However, certain live queries have a performance tax and can cause UI complications if overused, and it’s hard to turn them off in Meteor when you don’t need them. In REST, achieving live updating can be a difficult task where different parts of your UI can get out of sync. Wouldn’t it be great if you could turn reactivity on and off when you need it, with hooks for optimizing the performance at scale?
Query composition from the UI
To keep your code easy to read, you want UI components to be able to define their own data requirements, but aggregate those into one efficient data fetch from the client. We love the way Relay has achieved this, and think it is the right mental model for maintainable UI code.
What we’re building
Here’s what we are planning to build:
- Simple and efficient bindings for GraphQL that make it easy to publish data from REST services, SQL, and MongoDB.
- Reactive GraphQL queries that track changes to their underlying data, so clients can find out when the results have changed.
- Easy client-side query aggregation and fetching so that you can compose and run reactive GraphQL queries on the client, regardless of your view technology - Blaze, React, or Angular.
- Optimistic UI support based on lessons learned from 4 years of building optimistic systems within Meteor.
- Low-level hooks and options for analyzing and optimizing your queries that make it straightforward to scale your app.
Incremental and modular
Our goal is to build everything incrementally, so that every intermediate step is useful, and make the system as modular as possible so that the technology can be used in all kinds of tools, while still having a cohesive experience. We want to make something that is approachable and fast to develop with, while still having the tools to optimize your backend for scale.
One of our main goals is to make this new system easy to integrate with existing API technologies and view technologies. This means that while we are developing the new system, you should feel confident to keep using current methods and know that the migration will be easy.
- If you have an existing Meteor app with subscriptions and methods over DDP, you will be able to use them side-by-side with a Meteor Data GraphQL endpoint. This can be a good option to add SQL to your app, or to use a mix of reactive and non-reactive queries. In the long run, we expect that the new system can be a complete and straightforward replacement for Meteor’s current pub/sub mechanism, with lots of the new features that Meteor users have been asking for.
- If you have an existing Rails/PHP/Spring/Node app with REST endpoints, you can put a Meteor Data GraphQL server in front of it, so that you can use a GraphQL client to access the same endpoints but through a nicer, more performant API.
- If you are using a client-side data management system such as Redux, it will be easy to integrate server data loaded using Meteor Data.
What this means for you today
Follow along and pitch in to the discussion. We envision a system that will be incrementally adoptable when and where you need it, so you don’t have to worry about rewriting any of your already working code. If the project reaches a stage that is useful to you, please give it a try and let us know how it goes.
We are going to be working with people that have existing complex production applications written with Meteor and other platforms to ensure that adopting the new system is as clear and smooth as possible, and doesn’t cause undue fatigue for productivity-conscious developers. We will also make sure the the low-boilerplate getting-started experience available with Meteor today remains that way, so you’ll be able to build your prototypes just as quickly as before.
We’re excited to hear what you think of this idea, and what you have learned about application data loading in the last few years. We’re still working on the details of our design and implementation plan, and will post about it soon. In the meantime, please file issues on our Meteor Data GitHub repository or start discussions on the Meteor Forums. See you there!