Due to the nature of this application (virtual machine management), this could result in catastrophic data loss. It's easier to scale a database than a directory full of filesįlat files have a risk of someone copying a 'done' file back to the 'watch' directory.

The YAGNI principle states that flat files work just fine right now, there's no demonstrated need for changing to a more complicated solution, so leave it. This reduces the overall complexity of the code base and the amount of third party code that must be brought in. No queries to write, no risk of accidentally popping a message off the queue and having it be gone forever.įile management code is simpler than database APIs from a programming standpoint, since it's part of every language's standard library. There's zero risk of messages disappearing barring very low level bugs which would break other things anyways.įlat files require less technical sophistication to understand - just cat it. The arguments I've heard break down as followsįlat files are more reliable than any other solution, since the file only gets moved from a 'watch' folder, to a 'processing' folder after it's picked up, and finally to a 'done' folder when finished. It should be noted that Redis is used elsewhere in the organization for queued message handling. The argument comes in when one developer suggests that using the filesystem as a messaging layer is a bad solution, when something such as a relational database (MySQL), noSQL database (Redis), or even a plain REST API call should be used instead. The backend processing step takes about 10 minutes per message. all string data, no nesting), and around 1-2k at their largest, with the system spending most of its time idle (but bursting up to 100 messages at any given time). The files themselves are very simple (i.e. The back end polls this folder every few seconds, picks the file up, and carries out its functions.

json file) into a watch folder used by the back end. The web layer collects information by a simple web form, stashes this data as a JSON document (literally a. I've got an application which has generated a rather heated discussion between a couple of the developers.īasically, it's split into a web layer and a backend layer.

Relational database vs flat file