Maestro is a workflow module for Drupal which is in development by Nextide. We have created a design spec for this application, which we would like to share with the Drupal community for feedback. Maestro will be based on an existing workflow application Nextide has developed on another OS platform. Rather than a simply porting over the application, we would like to take the right steps to ensure it is properly integrated into Drupal.
For a quick 2 minute introduction of what Maestro is, check out our video
1. Maestro Overview
Workflow for CMS can be defined as the publishing of site content but there are also many business processes that can be automated and better managed via workflow like expense submissions to complex business processes such as new client provisioning, change management systems and work order approvals. While there is a clear need for solutions to provide hooks or triggers to existing Drupal content management data types and processes, we feel there is also a need for a solution that can define and manage complex business workflows. These can involve the routing of forms and attachments among task owners for different levels of approval, and include route changes depending on form data or other real-time routing control decisions.
Over the past 4 years, Nextide has developed a series of tools and applications for another PHP, Open Source framework which is used to manage and automate complex business workflows. We want to adopt and migrate these applications to Drupal and in doing so have been investigating what's available now for Drupal, and how best to integrate our solution into the Drupal framework as a native, best-in-class workflow module. Our other contributed module, filedepot, is an example of a tool that was developed in a similar manner over several years on another framework and which we have since ported to Drupal.
The existing workflow modules on Drupal provide a number of great solutions and show how flexible Drupal is and easy to extend with it's triggers and actions. These are features that we definitely want to include, however our solution goes beyond what is currently available. What if you could use a visual graphics workflow application to drag and drop a series of different task types with 'OR' & 'AND' type connectors to model your business workflow? These tasks could be: fill out a form, review and approve followed by conditional test task type that would route the workflow back to the originator if rejected with comments or move the workflow forward to another series of tasks.
Users will have a task console to let them manage their assigned tasks but will also receive notifications for new tasks and reminders. Supervisors can automatically be notified if tasks are past due and can review where in the process the workflow is. What tasks are outstanding, re-assign tasks, review forms, attachments and comments associated with the workflow instance.
There are a number of administrative functions that allow workflow admins to get detailed information on running and completed workflow instances, including process tracing. Workflow admins can re-assign tasks, send reminders and look at other users task console. If needed, workflow admins can execute tasks on their behalf and users can also setup their 'away' status and automatically re-assign new tasks to their delegate.
The workflow engine maintains statistics on all the tasks (assigned time, started time and completed time) so that it's possible to report on elapsed time for tasks, complete workflow instances. Managers really like to know what the average and worst case turn around times are for dept ABC to complete certain tasks.
The workflow admin maps out the workflow using the workflow editor creating a template. It will be possible for a workflow template to link or trigger other workflows. The single template or combination in case of more complex flows are defined as a project type. Each instance of a project type is tracked as complete project and all related tasks, forms and comments are viewable.
An example of a project is a 'New Employee Hire' or 'Office Supplies Order'. Users initiate a new workflow of type 'Office Supplies Order' and it would have tasks assigned to 'Supervisor' and 'Office Manager' roles. If the amount is greater the dollar amount then it requires the Department VP approval. The order is received and the initiator is notified. Complete workflow history and data is tracked and available for all that need to view and report on.
Additionally, it is worth mentioning that the tasks executed in a workflow are not advanced by user clicks, they are advanced by a task orchestrator which runs in your system behind the scenes. One of the main reasons for this is to ensure scalability in a site that runs thousands of projects and tens of thousands of tasks. The orchestrator handles all behind the scenes logic, such as sending a task notification or processing system tasks.
1.1 How Can the Drupal Community Help
The purpose of this document is to engage the Drupal community before we fully define the specification or start the module development. We need to be careful about scope creep, but we have much already available to migrate. Our experience migrating the filedepot module is that this will be at minimum a few man months of effort. We want to ensure that we include any key core Drupal integration features in the first release and plan for design items that need to be in the post 1.0 release. Identifying and prioritizing these requirements is where we would really appreciate feedback from the Drupal community so we can can have the best possible end result.
Our existing workflow solution is called Nexflow and will be known as Maestro when we port it over to Drupal as a module. We do not want to just port Nexflow over as it exists, it's important for the new module to be fully integrated and take advantage of the Drupal functionality. We have looked at a number of current Drupal workflow modules which all lack the full capabilities of Maestro and ultimately a true workflow automation solution. The key areas we think would be good integration points are actions/rules modules, CCK and the webform module.
In addition to porting and integration with Drupal, Maestro gives us the chance to take a step back and look at the Nexflow application improve its underlying framework. We have had Nexflow in production for several years with 200,000 tasks executed with 1000s of forms. After watching Nexflow in production environments we still see areas for improvement that are now possible when porting it to the new environment.
In summary, the goal with Maestro is to port Nexflow to the Drupal environment, integrate it fully, and improve upon the architecture to create a flexible and fully robust work automation system.
1.2 Drupal Integration
There are core areas that Maestro will integrate with Drupal. These areas include actions/rules, Drupal API hooks, as well as nodes, content types and taxonomy. The design of how Maestro's design uses Nodes, Content Types and Taxonomy is still in flux and an area we are looking for more Drupal development insights.
A workflow starts off life initially as a template created by the Template Editor. A workflow template will consist of many different types of tasks as well as process variables. Once defined, a workflow automates a specific process, anything from a capital request to adding a new user to the system. A project on the other hand, consists of many different workflows. A project also houses project-central data such as project resources, a project numbering scheme, and distribution emailing. With a project you can define that certain workflows must be completed before a new workflow can start, such as a project kick-off workflow, followed by the project closure workflow. Using Drupal's taxonomy, the related workflows will be linked to as projects. Its also worth noting that one workflow can be used in two different projects, sometimes can be executed more than once within a project, example: a request for information workflow.
- Can we manage this relationship using taxonomy or will we need to maintain these relationships in our own module tables?
The current thoughts are that workflow instances will be nodes and related workflow instances are grouped into projects using taxonomy. Each workflow template is unique and describes the workflow tasks and task routing with connectors and conditional logic.
- Should a workflow template be a content type or not? We think they need to be and that each instance of the workflow instance is a node. We can then link other related workflow content like forms, comments, attachments and even other linked workflow instances (as in a multi-workflow process) together using taxonomy.
What other Drupal data model related strategies should be considered in the design of this module? Most sites may only have a few workflow templates but our largest client would have about 30 workflow templates.
- Would that be difficult to manage if these were individual content types?
- Should workflow templates be nodes and not content types?
- If workflow template are nodes then how are we relating workflow instances once a user starts a new process - do we use taxonomy for that?
- If workflow instances are not nodes, how do we link in all related content like forms, attachments - do we just use module specific tables. With workflow instances being unique notes of a content_type, we have a nice high level Drupal data structure.
- What are the pros and cons that we should be aware of an considering when developing this high level data model / Drupal module architecture?
Drilling down to the task level, we do not see individual tasks being nodes but would be related data maintained in our module tables.
- What would the benefit be to site admin's for tasks to be nodes?
As part of the task definition, there would be a number of task properties that can be defined such as:
- Task Owner - assigned by role, username or process variable
- Task Notifications on assignment, completion and escalation rules
- Task type specific settings
We also see an opportunity to hook in actions and/or rules modules at the task level. When defining a task, it will be possible to set an action to launch on task creation or task completion. In addition, there will be a new task type added which simply wait for a trigger to occur. This will provide the ability for the workflow to automatically realize system events have occurred, such as a new page being added.
1.3 Existing Drupal Workflow Modules
We are not experienced enough with the existing workflow modules to know all of their strengths and recognize that there are work arounds to achieve almost anything. Perhaps other community members can add their comments and ideas based on use cases that they have deployed so that we are presenting our comments in a fair light. However, at this time there appears to be some key differences between these modules and what we are proposing with Maestro.
One consistent difference is that the idea of assigning a user a task does not exist, instead site users do something which triggers and action to occur. With Maestro, site users will initiate a new process (example request office supplies), and then other site users will be assigned tasks to complete in the order required. Users assigned tasks will be able to access them from their task console and may have only 1 task or possibly many for various other and different workflow instances. The types of the tasks in the other modules are essentially to complete a content type. With Maestro other task types exist such as to review submissions, approve or reject submissions, or answer other interactive questions such as assigning subsequent workflow step owners and changing the workflow routing. Workflow admin's have the ability to cancel workflow instances (projects) or re-assign task owners.
While all three of the modules compared in this section are quite powerful, there is a common drawback with all of them that really separates Maestro. The main difference between Maestro and the other modules can be summed up in one line: rather than waiting for site actions to occur and acting on them, Maestro drives the site actions to occur.
1.3.1 Workflow
The Workflow module, when combined with the Actions module, has the ability to create basic workflows. It can create "pseudo-tasks", which are represented by the state of the workflow. For example, if the task was to create a form, the state of the workflow would be "Create a Form", rather an a generic "Draft" or "Initiated" state.
The Workflow module integrates well with the Actions module, which is almost required to create a working process. Since workflows are built around content types in the system, the workflow module is well suited to facilitate site content creation.
Unfortunately, as we will see with the other modules, the Workflow module is based more around site actions and site content types rather than defining standard processes that get executed. Instead of a specific user being assigned complete a task as part of their job, it is up to the user to create a content type in the system, at which point the system will direct the lifecycle of that particular content type.
Maestro aims to build business project information, where multiple content pieces come together to form a project. Arbitrary documents can be uploaded and site users are actively assigned tasks which they can easily see in their task console. While Maestro will integrate with site content types, its purpose is not to build site content, but rather facilitate a company's procedure to complete work.
The Workflow module also lacks in conditional task abilities, it does not have any metric reporting, and has a clunky interface to create a workflow, which are all areas that Maestro will improve upon as well.
1.3.2 Workflow NG
Workflow NG is a more developed workflow solution which can handle certain conditional logic, and integrates well with other modules. However, it is no longer supported for future releases of Drupal as it has migrated to the Rules module. All of its strengths and weaknesses are similar to the Rules module, so lets leave it at that and jump straight into the Rules module comparison.
1.3.3 Rules
The Rules module is a great module to track system events and act on them. You can build up a workflow based on system events, so, for example, a form is created, that event is trapped and signals the next task to be created. You can define a ruleset in the Rules module which allows you to group related rules and execute them in order, to simulate a workflow. Rules is well suited to define a workflow to built site content and to act on site events.
However, similar to the workflow module, it does not support the creation of a standard "task", such as requiring a user to complete an action. Like the Workflow module, it waits for an action to happen, such as a new site content to be created, then performs actions on that, where as with Maestro, a process is defined to assign site users to create the content, and metrics are stored to track how long certain tasks have taken to execute.
With rules, there is no way to know what tasks are assigned to you, unless you set up an action to email the task owner on task assignment. The lack of a task console and a project dashboard page is another area that Rules lacks that Maestro will improve upon. Again, like with the Workflow module, Rules relies on system content types as the means of executing a workflow, where Maestro helps facilitate the building of business project information.
1.4 Planned Integration with Drupal Modules
1.4.1 Actions
Integration with the actions module is key. There are a number of different opportunities to integrate with the Actions module. Number one, we plan to include an action that launches a new workflow when trigger is fired. That will allow, for example, when a site user is created, launch the workflow that defines all the things that need to happen when adding a new employee to your company, like setting up their email address, etc.
Another key opportunity for integration is inside each different task type. On both task creation and task completion we will have the opportunity to execute an action, such as sending an email, or publishing a content type. The main purpose of this would be to send task notifications to users, however by integrating it into an action it makes that extensible.
The final area of integration into actions will is that we will have a task type that waits for a trigger to occur. So, if the workflow requires a user to be created, then the "new user created" trigger is fired, the workflow will continue on.
1.4.2 Tokens
For email notifications, tokens will be used to change out values like username, etc. It's important to be able to send out email notifications for task assignment but also send out task reminders, task alerts as to an identified contact for escalation if task not completed in x days and also task completion notifications. Our present workflow solution allows these different notifications and notification rules like sending the initial task reminder after 5 days and then every day, send an escalation to the initiator of the workflow after 10 days if task not completed. Being able to customize these notification email messages using tokens will be supported.
1.4.3 Rules
Maestro will be initially lightly integrated into Rules, where there will be an opportunity within Rules to create a rule which launches a new workflow. That again will allow a workflow to be launched after a specific site action has occurred.
In Closing
We would like to include the Drupal community in the development of this application, and include your ideas and feature requests as well. While initially we want to get this up and running in Drupal all feature ideas and feedback on how to properly integrate into Drupal will be considered in the initial port, and subsequently implemented in future releases.
This post contains the initial section of the Maestro spec and our plan is to publish the additional sections over the coming weeks which will get into more technical detail. That detail though is dependent on our initial high level spec and ideas/questions proposed in this document and why we are anxious for community input.
The Nextide Team
Focusing on Business Applications but heck we do CMS and Social Networking projects too
www.nextide.ca