Some time ago I've watched great presentation by Bret Victor called 'The Future of Programming' (it's still available on Youtube). This presentation is a great eye opener and got me thinking that maybe some things can be done in a different way. Bret said that it's impossible that 40 years from now (he pretended to live in early 70's) we would still write code in text editors. One of the great inventions at 'his' time was Flow Based Programming - an idea that application is a network of modules ('black boxes') exchanging messages. Given module should be stateless and for the same input data it should always return the same results. Sounds like a great idea but, for some reason, it didn't catch up (and we still write code in text editors). I decided to try it nevertheless and see for myself if this Flow Based Programming is of any use. I (Paweł Murawski) joined forces with Mateusz Zieliński and together we used our 20% time allocation to write an application using FBP paradigm.
Flow Based Programming
FBP seems like a great way to create application logic, but not necessary the application frontend, so we had to come up with a suitable project. Almost instantly we decided to create a synchronizer similar to one of our products - TFS4JIRA. It seemed perfect because it involves a lot of data retrieving and manipulation, exactly what FBP is good for. At first, we tried NoFlo - FBP implementation in NodeJS. To be honest, to this day I have no idea how to write any application in that technology. Fortunately, a few days later we found Node-RED. This project was also based on NodeJS but it actually worked as expected. We tried to write a simple Trello <-> JIRA Synchronizer and it took us only two days to accomplish that.
The best way to see Node-RED in action is to watch this video.
Creating our own additions to Node-RED
When we started working with Node-RED we missed editing Function nodes in our IDE of choice - IntelliJ IDEA. There were many reasons for that: code refactoring, code coloring, shortcuts - you name it. So we started to work on our own Function node. This node was able to read file from disk and this file could be edited in IDEA. Also we added functionality to check Function's input data. This took quite some time to accomplish but we didn't stop there. We created our own custom node for MongoDB connections, and then our own error handling system. So instead of creating new features and functionalities we ended up creating our own framework of sorts. After some time we created so much stuff that it was hard to come back to work on this project after one week break - we couldn't work with it all the time because it was only a 20% project.
Working together on a project
It's nice to work together on a project. You can simultaneously edit the same files with your colleagues and then merge all of the changes. But not in Node-RED unfortunately. A file that describes your network is just a JSON file full of data describing node coordinates and connections. It's next to impossible to merge changes of this file in text editor. Only visual merge tool would of a help here but, for now, there is no such thing. While working on a same part of a project (same network) it's just best to sit down next to your peer and connect those nodes together. True spirit of pair programming. Also introducing new people in a project would be quite a painful process because there are no good practicess and common knowledge of how to work on such an application.
Changes in big projects
While looking at big network of modules it's quite easy to deduct how the whole system works and what is the logic behind it, but the problem is that you cannot see the underlying model layer. It's created dynamically in many nodes and it's hard to reason what it looks like (and what it should look like) in any given place of the network. The only way to find out what data is available and when is to start an application and debug it. Also proper refactoring is next to impossible because changing name of some property in a model in one node could potentially create a bug in some other part of the system, because that other part of the system was expecting this property to have its old name. Refactoring in IDEA helps a little bit with that but still you need to remember to give very descriptive names, you can be sure that changing tha name of a property called for example 'project' would be quite a painful process.
Error checking and debugging
There are many places where error can happen: Node-RED, NodeJS, custom modules etc. This is quite confusing as you could search for an error in your browser, IDE (both of them: IDEA and Node-RE) and console window where NodeJS application was started. We tried to even create our own error handling system but after two weeks we didn't even remember how it works. As for debugging - it really comes down to connect Debug node to already existing network and check what data goes into it. It involves changing the network structure and redeployment of application. Good thing that this redeployment takes fracture of a second (really - it's that fast, this is huge time saver).
This is more of a NodeJS problem than Node-RED, but still I want to write a few words about it. We used NPM to manage dependencies in a project and I have to say that I cannot understand who and for what reason decided to have operators in a dependencies definitions. One of the libraries that we used in a project was dependent on some other library in a version bigger than 1.0. Of course this blowed up on a production, because meanwhile that other library was released in a version 2.0 with its API completely changed. I can't understand why people just don't use fixed version of a dependency, what's the reason to use operators? Especially "bigger than", which is like the worst idea ever.
Too many new technologies
Because this was our 20% project we wanted to learn as much as possible. We decided to introduce in this project many technologies we didn't know or wanted to know more of: Node-RED, ReactJS & Flux, Trello, MongoDB, NodeJS. After some time we realized it was too much - instead of finishing tasks we were constantly learning. I think that at least for some part of the system we should have used technologies that we were more experienced.
Deployment of this application was interesting experience to say at least. After few tries we finally hosted it on a Heroku as 3 separate applications (Connect Plugin, Synchronizer, REST API) and the MongoDB database was on some other hosting - the one that didn't require us to provide credit card data. Setting this whole things up and finaly showing it in JIRA Cloud was very satisfying achievement.
First version of Trello4JIRA is deployed and it works but it lacks many features. This project started because of pure technological reasons but now it has to be assessed from the business point of view. We stopped working on it until somebody decides that going further with it is financially sensible . We've learned a lot and for now this is the biggest gain from this project (not counting this blogpost ).
Bird's eye view of the finished Synchronizer.
Conclusions about FBP
FBP, in theory, is a great way to create some type of applications, especially the ones that have to retrieve a lot of data from different systems and transform them. One of the reasons for creating Synchronizer using FBP technology was to check if it's better way to work on a software than the "classical" way. The problem is that there are not too many good implementations of it. We decided to use Node-RED because it seemed quite powerful and relatively easy to use. So was it better than just writing all this Sync stuff in Java? In my opinion not right now, but it could be in the future. Currently it lacks some very important features like strict definition of a model layer, support for better refactoring and possibility to work in teams. Als good practices has yet to be emerged, not only in code itself but also in network visual node organization. In my opinion Node-RED is very good for small hobby projects and that's all, I wouldn't recommend using it in a big project. Maybe it will be in a few years from now - it's worth keeping eye on it.
What we've learned
- A few things about FBP and what it's good for (and what it lacks)
- New technological skills: Node-RED, NodeJS, ReactJS & Flux, Atlassian Connect NodeJS bootstrap project, Trello API, MongoDB
- That you should always try to cut the scope. Really - always.
- Creating your own framework takes much more time than one can anticipate. And sometimes it just doesn't make sense.
- 20% projects should be small. This was too big fish to fry in such a small amount of time.
- When starting new project it's good to use at least a few technologies that you already know.
- It's really fun to connect those nodes.