reddit answer on workflow engines

Here is an answer I wrote for go to resource on how to build a workflow engine.

Robert was asking:

So I'm researching how to build a workflow engine, is there like a "standard work" or "go to resource" for this topic? Just starting out, will keep a log here.

That makes for a cheap blog post, but I'd like to keep that as an archive.

my answer

I built three workflow engines over the years. I have never found a "how to build a workflow engine" resource, but I found workflow engines (all of them proprietary when I started), research papers about workflow modelling, budding industry standards, and blogs.

My idea of a workflow engine, then and now, was of a tool that accepts a model of a business process (a workflow definition), something rather high-level describing the flow of work among participants to produce a business result. The developers and the business people should be able to gather around such a document and agree on it (later on, iterate on it to adapt to new situations).

The document/model/piece of code should rally business logic, instead of having it scattered all over the place.

For the two last engines I built, I used the Workflow Patterns to guide me, more specifically, the Control-Flow Patterns. A workflow engine, in my opinion, has to cover the principal patterns listed there to be of any use.

Another resource I came to appreciate, is BPMN. It's a business process model and notation. It is mostly used to document business processes and a subset of it is destined for "execution". I have seen organizations use BPMN to document "manual" business processes "as is", and sometimes to model business processes for execution by a workflow engine "to be".

BPMN is important because more business people are getting exposed to it. They are attracted like flies to the promise of drawing executable diagrams. Ten years ago, the buzzword was "no code". But then, recently, it became "low code". It's hard to abstract away the programming.

My favourite blog about BPMN and its surroundings is Process Is The Main Thing.

It might be necessary to look the series of blog posts culminating in Business Process Models are Not Agile for a rebuttal of the grail of business process modelling.

It seems that many times, a box and arrow system is too top-down to succeed. It'd be better to have a kind of task list with termination and delegation rules/constraints, and tracing. The balance is yours to find.

A task / worklist system at the front, with a workflow engine behind driving some of the non-loose business processes might make a good combination. When building task lists I tend to look at Workflow Resource Patterns, the resource subset of the patterns mentioned above.

You could start with a tasklist and as the time passes, iterate towards having the repetitive task patterns codified into business processes, cataloged in a process portfolio, itself iterating as the business conditions evolve.

You will also come across the term "workflow engine", but that'll cover scientific workflow engines as well, those are more specialized into the process of data workflows. From the other answers here, I guess you are more interested into "business process" engines. Googling for "Ruby workflow" will lead you in world of development techniques.

So, the resources I have found and used are mostly about what to build, not how to build it. As a programmer, my customers usually tell me what to build and the how to is for me to figure.

My two Ruby workflow engines are ruote which is now dead, and its successor flor. They are web framework agnostic. For the task/worklist companion to flor, I am building florist.

In the flor readme, you'll find a list of other Ruby (and Rails) workflow engines.