Open Sourcing DataFire

I’m excited to announce that we’re releasing the next version of our integration framework as open source software…it’s like Grunt for APIs, or Zapier for the command line. You can try out a preview — released under the MIT license — on GitHub.

Background

For the last year, DataFire has provided a platform for building, running, and managing integrations. As cloud software continues to eat the world, valuable, actionable data is often trapped in SaaS silos, and getting services like Slack, MailChimp, and Salesforce to work together, as well as with your own services and databases, can create a tremendous amount of value. While platforms like Zapier and IFTTT provide beautiful GUIs for building and sharing simple integrations, they often lack the power and expressiveness of good old-fashioned code. DataFire sought to fill this gap by allowing users to write snippets of JavaScript that control how data is retrieved and transformed.

Falling Short

However, we quickly found that what we built still wasn’t powerful enough. Sure, it was now much easier to manipulate data in transit, but there was no way to pull in external libraries or reuse code between flows. Furthermore, coding inside the browser was a painful and foreign experience, sort of like trying to ride a bike on sand. And our solution to version control — exporting flows to GitHub gists — was less than optimal.

With the new open source release, all that changes. You now have access to the entire NodeJS ecosystem, including every package in npm. You can easily factor your code into multiple files and share code across projects. Flows can live inside any version control system, and you can work in the editor of your choice.

Even more importantly, we’ve decoupled flows from the DataFire platform. While you can still run and manage flows on datafire.io, you can also run them locally, or on AWS Lambda, Google Cloud, and Microsoft Azure via the Serverless framework. We believe it’s critical that open source software be truly open.

Open vs. Proprietary

It’s become starkly clear in recent years that certain problems are best solved by communities, while others are best solved by organizations. Version control, for instance, needs to be immensely general and flexible in order to accommodate the rapidly growing number of programming languages, as well as the rich variety of organizational structures that rely on it. Had Linus Torvalds made an uncharacteristic attempt to monetize Git, it would have quickly specialized in whatever the highest bidders needed — probably Java and C++ code management. And it would have eventually been supplanted by a more general version control system with a larger, democratic community of stakeholders.

Social networks and GUIs, on the other hand, need a more top-down, autocratic approach. They need to articulate and maintain a clear vision — trying to be everything to everyone will only dilute the experience and alienate users. The obvious example here is GitHub — they’ve succeeded by creating a simple, beautiful user interface around a complex piece of software, and by providing a space that catalyzes collaboration. GitLab is challenging this with an open-source alternative, as well as a more democratic approach to feature development, but GitHub has the clearer vision, and in terms of users and mindshare, GitHub is winning.

We believe integration frameworks fall squarely in the open source bucket. Almost by definition, they need to be everything to everyone. I can’t begin to tell you how many weird, unexpected use cases we’ve encountered, or of all the strange ways services have of exposing their data to the world. In order to accommodate and prioritize amongst the immense, evolving variety of data formats and transfer protocols, it’s critical that an integration framework be directed by the community of its users. We hope that the open source release will encourage users to contribute ideas, opinions, and code, so that DataFire can evolve alongside the people and organizations it serves.

Free as in Freedom

While many users will choose to run flows on their own infrastructure, our overall business model remains unchanged. Self-hosting comes with the unavoidable burden of managing servers and monitoring logs — DataFire.io will provide a much friendlier interface for running, managing, monitoring, and logging flows, and the backend will be more robust and scalable than a cron job.

We are encouraged by the massive success of other companies that have built products around open source software, and know that as we create a sustainable business around DataFire.io it will serve to strengthen the open source framework.

Moving Forward

Our primary goal in open sourcing DataFire is to empower our users. Allowing you to work in the environment of your choice will be far more comfortable and productive. And providing access to data, not just for our 250+ native integrations, but no matter where it lives, makes the framework an order of magnitude more powerful. v2 also introduces a number of new exciting features, such as error handling, asynchronous steps, and pagination support, which we’ll be writing about in the coming weeks — stay tuned!

For now, the best way to help is to get involved. Try out the the framework, read over the docs, and let us know what you think!

Hacker Noon is how hackers start their afternoons. We’re a part of the @AMIfamily. We are now accepting submissions and happy to discuss advertising &sponsorship opportunities.

To learn more, read our about page, like/message us on Facebook, or simply, tweet/DM @HackerNoon.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!