The Pluggable Distibuted Data Processing Daemon
The PDDPd allows creation of a customized system to process data in a distributed environment. The system follows a simple yet flexible data processing model, allowing custom code to be inserted dynamically at each stage of processing. The system is designed to be remotely administrable.
Note: I plan to continue working on this project, but have no time to do so at this moment. If you are interested in this project, or a particular feature which has not been implemented yet, please use the Tracker from the Project Page to let me know.
The goal is to create an efficient, remotely administrable, and stable Daemon which will be given plugins to run. These plugins must obey certain criteria regarding dataflow, but their purpose can be anything. The Daemon's dataflow model is simple and flexible: Data is produced, data is processed, and data is consumed. Therefore, each Daemon has three plugin types (Producer, Processor, and Consumer) through which data flows. How data is produced, processed, and consumed is entirely up to the plugin developer. The Daemon (in addition to handling the loading and running of the plugins) has facilities to perform on-the-fly configuration of itself and the plugins (including loading new plugins from the network) and logging functionality. The software is written in Java.
The project consists of three parts:
