Flamenco, Computational Cluster Development Story Continued. Part 3

Last time we talked about Flamenco system and one of its key features peer-to-peer topology. And now I would like to explain how Flamenco met challenges:

• Cross-platform portability
• Dynamic scalability
• No external dependencies (under development)

Cross-platform portability

Due to the Java platform, the node Flamenco can be launched under any operating system from Windows to Solaris. It makes it possible to use Flamenco both on office computers (that are most often run on Windows), dedicated servers, and even on Linux-run cloud virtual machines. As a result, it opens up ample opportunities for flexible operation of the cluster capacity. Flamenco installer contains everything that is required (scripts, etc.) to launch an application in the console or to set it up as a full service.

Dynamic scalability

Basic functionality doesn’t contain support of the performance of any specific tasks – just control over them. Each task is implemented separately and can be transferred to Flamenco for performance by means of plug-in mechanisms. Plug-ins can be connected and disconnected dynamically. A remote repository available in the HTTP protocol is used for their storage. When a new task is given to the node, a check-up of support of this functionality starts. If such check-up is missing, a search of the corresponding plug-in in a repository with its subsequent installation is carried out. After that, the task is implemented. Thus, support of a new functionality can be added on-the-fly, without cluster restart.

Absence of external dependencies (under development)

Absence of dependencies on external coordinators, such as Apache Zookeeper. It allows simplifying cluster setup since there is no need in additional installation and support of external services (as well as providing fault- tolerance). At present, there is dependency on the external file system (access is provided under the FTP protocol), that in the future is planned to be replaced by the internal one. The relative experiments with the internal file storage have already been carried out, but at the moment this component cannot become a full replacement of the available decision, and it requires additional researches and rework.

To be continued…

Previous parts

Flamenco, Computational Cluster. Part 1: Introduction

Flamenco, Computational Cluster. Part 2: Flamenco. Peer-To-Peer Topology