jmettraux.skepti.ch

20190407 flor workflow engine en flor workflow ruby

flor is a workflow engine. It's written in Ruby.

A workflow engine is a host for workflow executions, business processes that might last from a few minutes to multiple years.

Such an engine takes as input workflow definitions and an initial set of data and controls the execution "flow" until success or failure.

tl&dr

Look at flor's readme, at its one Ruby file doc/quickstart0, or at its more frameworky complex doc/quickstart1.

workflow definitions

At its core, flor is an interpreter for the flor workflow language. The "work" in "workflow" is assumed by taskers. Flor and its language deal with the control flow between taskers.

Flor doesn't use a Ruby DSL or a BPMN diagram to represent its workflows. The flor language is a programming language dedicated to workflows. It has its advantages and disadvantages, as with all pieces of software, we shall hopefully iterate towards a language that covers a large part of its community needs.

Here is an example reporting workflow, it involves five human participants and two automated taskers:

alice 'prepare feedback forms'
concurrence
  eric 'gather customer feedback' timeout: '8d'
  fred 'gather customer feedback' timeout: '8d'
  greg 'gather customer feedback' timeout: '8d'
anonymize_feedback _
loop
  alice 'prepare report'
  brent 'validate report'
  break _ if f.valid
submit_report _
brent 'close report'

This example of workflow definition routes work from Alice, fans out to Eric, Fred, and Greg, giving them each 8 days to gather customer feedback. Once all have replied (or 8 days have elapsed), the feedback is anonymized. A back and forth between Alice and Brent is started, until Brent deems the report ready for submission (by setting the field valid to true). Finally Brent closes the report.

The flow definition weaves in calls to taskers, moving tasks from one to the next.

Any number of workflow executions can coexist in a flor instance. A workflow execution in a given version can coexist with a workflow execution following the same definition or any other definition. Executions following the same definition but in different versions can coexist.

taskers

A tasker is a piece of Ruby code that is called to perform a task. Here is the code behing the anonymize_feedback _ above:

class Acme::AnonymizeFeedbackTasker < Flor::BasicTasker

  def task

    Acme::Models::Forms   #
      .where(exid: exid)  #
      .each(&:anonymize)  # access DB (Sequel style) and anonymize each
      .each(&:save)       # form related to this execution (exid)

    reply                 # hand back task to flor
  end
end

The flor language and the taskers try to be orthogonal. Taskers should be easy to test on their own and be usable from multiple workflow definitions.

When flor is set up with a HashLoader, it's fairly easy to directly bind tasker implementations to the names appearing in the workflow definition:

FLOR = Flor::Unit.new(
  loader: Flor::HashLoader,
  sto_uri: 'postgres://flor:xxx@localhost/flor')

%w[ alice eric fred greg brent ].each do |user|
  FLOR.add_tasker(user, Acme::HumanTasker)
end
FLOR.add_tasker(:anonymize_feedback, Acme::AnonymizeFeedbackTasker)
FLOR.add_tasker(:submit_report, Acme::SubmitReportTasker)

Human taskers are usually placing the incoming task into a worklist, let users edit it and then, when they're done with it, return the task to flor. There is the florist project providing a vanilla flor worklist, but it's work in progress.

launching

In the case of a web application, the launch of a new workflow execution usually happens in a controller, here is an example Sinatra POST endpoint:

post '/xhr/reports/' do

  exid = FLOR.launch(
    REPORT_FLOW,
    payload: prepare_payload(request))

  content_type 'application/json'
  { exid: exid }.to_json
end

Behind the scene, that simply puts a launch message in flor's database and returns quickly the exid for the new workflow execution.

two kinds of configuration

More advanced flor deployments might favour the configuration tree approach demonstrated in flor/quickstart1/. Taskers and flows are placed in a tree hierarchy mimicking an organization's domain and subdomains. (What stands behind a subflow or a tasker might thus depend in which (sub-)domain the flow is launched).

The example in this post is based on the straightforward flor/quickstart0/ which spans a simple Ruby file.

two kinds of setup

The simplest way to deploy flor is within the web application it serves, in the same Ruby process. That's what I use in production, thanks to the wonderful JRuby.

A more classical setup is to have 1 or more Ruby processes dedicated to the web application and 1 or more Ruby processes for flor, a worker process. That's totally fine. The flor unit in the web processes can be instantiated without being started, while the flor unit in the worker processes is started. The flor units share the same database and communicate via it. (See doc/multi_instance.md).

flor characteristics

• Written in Ruby
• Not tied to any (web) framework
• Only one major dependency, the excellent Sequel

Ruby might be deemed a slow language, but it's fast enough when thinking of business processes spanning long periods of times.

flor features

• Has support for timeouts and time scheduling
• Workflows can be cancelled completely or in part
• Workflows can be re-applied with newer versions of part of their definition
• Errors handlers can be set at various levels
• Workflows can wait for signals
• etc

over

If you have any question, please ask in the flor Gitter room or via the GitHub issues.

20180927 the flor language en flor workflow ruby

Flor is a language for defining workflows for the flor workflow engine.

A flor engine is an interpreter for the flor language. Usually an interpreter lives in an OS process and interprets a single program. With a flor engine, there are multiple program instances, called workflow executions, being interpreted in the same engine. Flor can host multiple executions of the same workflow definition or executions of various, different, workflow definitions. They could be various versions of the same workflow definition, or even executions whose definitions diverged in-flight from their original definition (rewriting the program in-flight).

Executions route work between taskers. Taskers are orthogonal to workflow definitions/executions. Multiple definitions/executions share the same set of taskers. For example the "accounting department" tasker or the "send notification email" tasker may appear in multiple workflow definitions (in the same business domain usually).

Executions spend most of their time waiting for taskers' answers, especially if those taskers are human.

This meandering post tries to show what flor, the language, looks like, and to explain the motivations behind that. This post doesn't look at tasker implementation, it is sufficient to know that they are scripts (Ruby or not) accepting task messages and replying with an updated task payload or rejecting the task.

the language itself

Here is a sample flor workflow definition:

sequence

  set item_id                          # the launch data is used
    create_mandate _                   # to create a new mandate

  cursor

    ops 'assign container number'      # operations team
    update_mandate status: f.status    # ops decides on status

    break _ if f.status == 'numbered'  # break out of cursor if 'numbered'

    rm 'rework mandate'                # relationship manager
    update_mandate status: 'agreed'    # refines mandate data

    continue _                         # mandate goes back to operations...

  pfs 'activate mandate'               # portfolio team
  update_mandate status: 'active'      # activates the mandate

  email 'rm' 'awm mandate activated'   # success, email sent to rel manager

A few observations:

• It is indented
• There are no end of indentation levels, a leftward indentation change ends one or more indentation blocks, a rightward indentation change starts a new indentation block, it's like Python
• Lines begin with a word followed by arguments
• Arguments are strings or key: value entries, or sometimes _
• Comments start with a # sign and end with the end of their line
• The head of line word sometimes is an imperative (set, break, update_mandate, email, ...), sometimes a noun (sequence, rm, ...)
• Like in Ruby, some lines end in a condition, for example: break _ if f.status == 'numbered'

A few answers:

Each non-comment line is a "call" of the head-of-line word procedure

Some of those heads are common flor procedures (set, sequence, cursor, break, continue, ...)

Some of those heads are taskers (pieces of code registered in the flor engine and available to multiple flor definitions), here, for example, "rm", "ops" and "pfs" point to three distinct group in the organization using the flor engine, they are "human taskers"

Some of those heads are taskers pointing to non-human taskers, like "email" and "update_mandate", such taskers tend to perform their task immediately, while human taskers are usually delivering the task to a queue/list system where humans may pick them (and drop them back eventually)

Some of those heads could be functions (sorry, no function calls in the example above).

The suffix conditionals are in fact syntactic sugar:

break _ if f.status == 'numbered'
  #
  # gets rewritten to:
  #
if
  f.status == 'numbered'  # conditional
  break _                 # then branch
  # nada...               # else branch

Such rewrites occur when the flow reaches the break/if, not before.

The _ (underscore) as a single argument is necessary to distinguish getting the value of a head from calling it.

Consider:

define get-name
  "Hector"

get-name   # yields a pointer to the "get-name" function
get-name _ # yields "Hector"

Yielding a pointer to a function has its uses, for example, when aliasing it:

define add x y \ + x y

set plus add  # alias plus to add
plus 1 2      # yields 3

set plus      # set plus
  add 2 3     # to the value of `(add 2 3)`, yields 5
plus 3 4      # yields 5, not 7

I'm sure there are more questions, feel free to open a question/issue on the flor issue tracker or to drop in for a chat in the flor chat room.

why such a language?

Like all programming languages, flor sits between the user and the interpreter. It has to be understood by the interpreter (fairly easy) and by the user (difficult) and his co-workers (even more difficult).

Business people out there know and use BPMN. Why not use that?

Please note that I describe flor as a "workflow engine", not a "business process management suite". I am not really targetting business users, I am targetting fellow developers. Funnily, for most developpers, a workflow is in the realm of "my workflow", the set of ways in which they weave their work, all alone. Granted sometimes, there are team workflows, but there are much more blog posts about "my development workflow" out there than about team workflows.

Still, why not use BPMN? I prefer a text programming language to a visual programming language. I could go christian and say "In the beginning was the word", not the diagram. I also do not think that a diagram is not a program, there is no "no code", a workflow definition, text or diagram, is a piece of code. Most diagrams out there are backed by textual notations, we're back to word, text.

The basic blocks in a programming language come most of the time straight out of the English language: GOTO, if, car, cdr, RETF, whatever... We strive to have programs reading like English. If I target developers, I have not to forget that they'll be sharing those programs, those workflow definitions, with business users. It'd better be readable.

As already written, the flor language essentially routes work among taskers. Concise definitions should be readable. Throwing the work over the wall to a tasker should hide/abstract most of the complexity, then flor allows for functions, a classical way to abstract details away, flor then routes work among taskers and functions. The way a tasker is called is similar to the way a function is called: tasker name or function name, followed by arguments.

define phase1
  concurrence
    alice 'call A-N customers'
    bob 'call M-Z customers'
define phase2
  sequence
    alice 'get customer confirmation'
    bob 'do something else'

sequence
  phase1 _              # function call
  phase2 _              # function call
  charly 'final phase'  # tasker invocation

#verb_or_noun arguments*
#  verb_or_noun arguments*

I am not against diagrams, I think they are great to provide the coup d'oeil that business users want. My language should lend itself to (semi-)automatic generation of diagrams. If "technical details" are hidden behind abstractions, meaningful diagram generation should be possible.

Ruote, the predecessor to flor had a Javascript library for diagram generation. Flor should have one too.

It still takes discipline to write readable flow definitions or to write graspable BPMN diagrams.

a Lisp bastard

Flor is a Lisp bastard, one of those that lost the parentheses. A word on its own is replaced by the value it stands for. A word is followed by arguments and we have a function call or a tasker invocation, or simply a procedure call.

The 10th Greenspun rule needn't apply, I am deliberately bringing a Lisp to the table. Granted, "bug-ridden", "slow" and "half-complete" may still hold.

If Lisp is the mother, who's the father? Ruby for sure.

# Lispish
#
map [ 1, 2, 3 ]      #
  def elt            #
    + elt 3          # "map" accepts a function

# Rubyish
#
collect [ 1, 2, 3 ]  #
  + elt 3            # while "collect" is a macro rewriting to a "map"

# both yield [ 4, 5, 6 ]

Surely, using flor to add 3 to a sequence of integers is a waste. Here is maybe a better example:

define notify_warranty_void users
    #
  set emails                     #
    users                        #
    collect                      #
      elt.email                  # collect the email for each user into "emails"
  notify emails "warranty void"  # call to function "notify"
    #
    # OR
    #
  notify (users | collect \ elt.email) "warranty void"  # as a one-liner

There is a list of the flor procedures, sorry, it doesn't contain "car" or "cdr" for the time being.

Why a Lisp bastard for defining workflows? I could answer that it's because I got taught about programming with the Wizard Book, but it's rather my wish for a language close to its symbolic expressions. I want this (verb_or_noun arguments*) structure.

conclusion

It can be said that I have an obsession for workflow engines, maybe it's more of a quest for a workflow language.

Lots of business processes are weaving among participants and services. Some of them are described (as-is BPMN diagrams), lots of them are bizlore.

Some business processes are backed by state machines, but most of the time, those are lifecycle business processes, they concern 1 entity, there might be a higher level business process driving the lifecycle of multiple entities.

Some business processes can be executed advantageously by a workflow engine. Having a workflow definition mapping one-to-one to a workflow execution is great. The definition can be versioned, executions of one or more versions of the same definition may coexist.

A portfolio of concise business processes / workflow definitions is very important. It takes discipline.

I am trying to provide a fun language to build that with.

20171021 flor design 0 en flor ruby workflow

I was approached on Twitter by Benjamin Dauvergne asking me about flor's design.

@jmettraux est-ce qu'on pourra avoir un jour un article de blog sur le design de ruote/flor ? :) Je n'ai jamais rien vu d'équivalent.

— Benjamin Dauvergne (@b_dauvergne) October 5, 2017

Could we have one day a blog post on the design of ruote/flor? :) I never saw anything equivalent.

Flor is a Ruby workflow engine. It is a new iteration of ruote which is now dead.

Ou des pointeurs sur l'inspiration, j'ai toujours du mal avec l'exécuteur d'expressions/message, on dirait un design tuplespace/whiteboard.

— Benjamin Dauvergne (@b_dauvergne) October 5, 2017

Or pointers on the inspiration, I always had troubles with the expressions/message executor, it looks like a tuplespace/whiteboard design.

(An entry point on tuple spaces: https://en.wikipedia.org/wiki/Tuple_space)

I pointed at flor's nascent documentation, but, well, that isn't a blog post, it's an index, not a gate.

Vu, mais j'ai toujours du mal avec les exécutions, les noeuds et les messages. Il faudrait un exemple concret du début à la fin.

— Benjamin Dauvergne (@b_dauvergne) October 16, 2017

Seen, but I still have trouble with the executions, the nodes and the messages. A concrete example, going from start to finish, would help.

Tel exécution lance tel message, tel fichier JSON est posé, etc.. :) Désolé si j'abuse un peu, mais ça parait assez magique.

— Benjamin Dauvergne (@b_dauvergne) October 16, 2017

This execution emits that message, this JSON file is produced, etc... :) Sorry if I push too far, but that looks quite magical.

ruote or flor

Flor builds on ruote. I only want to use/discuss ruote's design in order to explain the design and the design decisions around flor.

Flor may be thought of as a shrinking of ruote, a simplification. A workflow engine must be extremely robust, at least it must let executions be fixed with relative ease. That shrinking / simplification should go in that way.

executions, nodes, messages

Benjamin is mentioning among others, those three flor concepts, executions, nodes and messages. There is a flor glossary, it's currently incomplete. Those three concepts deserve some kind of explanation.

An execution is an instance of a flow. A flow is a [business] process definition. There can be many executions of a flow. You create an execution by handing a launch message to a flor scheduler. A launch message is an "execute" message. The two main messages in flor are "execute" messages and "receive" messages. The execute messages ensure the buildup of nodes, the receive messages accompany the termination of nodes.

Here is an example flow:

sequence
  task 'alice' 'stage a'
  task 'bob' 'stage b'

Executing it, in other words, creating an execution of it, starts with an "execute" message, with the given flow, translated into a[n abstract syntax] tree, being handed to the flor scheduler.

The scheduler will create a temporary component, an executor to run the execution. The executor will consider the initial execute message and create the corresponding node with a "nid" of "0" (root node). The node is a storage artefact, the executor then hands message and node to the "sequence" procedure, where the behaviour of a sequence is determined (execute children 0, wait for its receive message, execute children 1, wait..., when there are no more children, send receive message to parent node).

Execute messages build up the execution (seen by the addition of nodes), while receive message are symptoms of the execution building down (usually receive messages go hand in hand with the executor removing nodes from the execution).

I write that the executor is a temporary component because it only lives for the time of a session, when there are no more messages to proceed, the executor ends. When new messages show up, the scheduler will create a new executor who will pick up the execution where the previous executor left it (in the database).

The vanilla example of a "session" would be, following the flow above, the run from the sequence to its first child, the task emitted to the 'alice' tasker. When the "task" message has been handed to alice, there are no more messages to proceed, the session ends, the executor finishes.

a [hopefully] concrete example

Thanks to Asciinema and kinocompact, here is a run of the above flow. When the executions happen, there are summaries of the messages emitted to stdout, I hope it's not too confusing.

I extracted the messages for the three sessions fo the "rucho" execution. The "exe" lines are for "execute" messages. The "rec" lines are for "receive" messages. The "tas" lines are for "task" messages. Each of the below "ucho" lines are summaries of a single message. Note the node ids, 0, 00, 01, etc...

Here is the first session, where the execution is launched. It ends with 2 nodes (sequence 0 and alice tasker 0_0) and a task handed to alice. There are multiple "tas" messages since the executor discusses with the ganger to determine what tasker should receive this task labelled "alice".

I had better call those sessions "runs", as you can see in the logs, that's the word flor uses.

    /--- run starts Flor::UnitExecutor 70115976477080 shell-cli-20171021.0830.mechulorucho
    |   {:thread=>70115976196240}
    |   {:counters=>{}, :nodes=>0, :size=>-1}
   ucho 0 exe [sequence L2] [task,[[_att,[[_sqs,alpha,3]],3],[_att,[[_sqs,"s... m1s_" f.ret null vars:var0,_path,root,name
   ucho 0_0 exe [task L3] [_att,[[_sqs,alpha,3]],3],[_att,[[_sqs,"stage a"... m2s1r1>1 from 0 f.ret null
   ucho 0_0_0 exe [_att L3] [_sqs,alpha,3] m3s2r1>1 from 0_0 f.ret null
   ucho 0_0_0_0 exe [_sqs L3] alpha m4s3r1>1 from 0_0_0 f.ret alpha
   ucho 0_0_0 rec [_att L3] hp:_att m5s4r1>1 from 0_0_0_0 f.ret alpha
   ucho 0_0 rec [task L3] hp:task m6s5r1>1 from 0_0_0 f.ret alpha
   ucho 0_0_1 exe [_att L3] [_sqs,"stage a",3] m7s6r1>1 from 0_0 f.ret alpha
   ucho 0_0_1_0 exe [_sqs L3] "stage a" m8s7r1>1 from 0_0_1 f.ret "stage a"
   ucho 0_0_1 rec [_att L3] hp:_att m9s8r1>1 from 0_0_1_0 f.ret "stage a"
   ucho 0_0 rec [task L3] hp:task m10s9r1>1 from 0_0_1 f.ret "stage a"
   ucho 0_0 tas [task L3] hp:task m11s10r1>1 from 0_0 f.ret "stage a"
   ucho 0_0 tas [task L3] hp:task m12s10r1>1 from 0_0 f.ret "stage a"
    |   run ends Flor::Logger 70115974783520 shell-cli-20171021.0830.mechulorucho
    |   {:started=>"2017-10-21T08:30:48.174253Z", :took=>0.019922}
    |   {:thread=>70115976196240, :consumed=>12, :traps=>0}
    |   {:counters=>{"runs"=>1, "msgs"=>12, "omsgs"=>0}, :nodes=>2, :size=>465}
    \--- .

The executor ends, its run over, alice has its task. The execution (the sum of nodes) has been saved to the database.

As you've seen in the asciinema above, I then return this task to the execution (a "return" message placed in the flor scheduler database). The scheduler notices it, instantiates an executor, which removes the alice task node, sends a "rec" message to the sequence, which then sends an "exe" message to its 0_1 child, the bob task.

    /--- run starts Flor::UnitExecutor 70115975852240 shell-cli-20171021.0830.mechulorucho
    |   {:thread=>70115975803940}
    |   {:counters=>{"runs"=>1, "msgs"=>12, "omsgs"=>0}, :nodes=>2, :size=>465}
   ucho 0_0 ret [task L3] hp:task m13s_ f.ret "stage a"
   ucho 0_0 rec [task L3] hp:task m14s_r2>2 f.ret "stage a"
   ucho 0 rec [sequence L2] hp:sequence m15s14r2>2 from 0_0 f.ret "stage a" vars:var0,_path,root,name
   ucho 0_1 exe [task L4] [_att,[[_sqs,bravo,4]],4],[_att,[[_sqs,"stage b"... m16s15r2>2 from 0 f.ret "stage a"
   ucho 0_1_0 exe [_att L4] [_sqs,bravo,4] m17s16r2>2 from 0_1 f.ret "stage a"
   ucho 0_1_0_0 exe [_sqs L4] bravo m18s17r2>2 from 0_1_0 f.ret bravo
   ucho 0_1_0 rec [_att L4] hp:_att m19s18r2>2 from 0_1_0_0 f.ret bravo
   ucho 0_1 rec [task L4] hp:task m20s19r2>2 from 0_1_0 f.ret bravo
   ucho 0_1_1 exe [_att L4] [_sqs,"stage b",4] m21s20r2>2 from 0_1 f.ret bravo
   ucho 0_1_1_0 exe [_sqs L4] "stage b" m22s21r2>2 from 0_1_1 f.ret "stage b"
   ucho 0_1_1 rec [_att L4] hp:_att m23s22r2>2 from 0_1_1_0 f.ret "stage b"
   ucho 0_1 rec [task L4] hp:task m24s23r2>2 from 0_1_1 f.ret "stage b"
   ucho 0_1 tas [task L4] hp:task m25s24r2>2 from 0_1 f.ret "stage b"
   ucho 0_1 tas [task L4] hp:task m26s24r2>2 from 0_1 f.ret "stage b"
    |   run ends Flor::Logger 70115974783520 shell-cli-20171021.0830.mechulorucho
    |   {:started=>"2017-10-21T08:36:50.418712Z", :took=>0.012553}
    |   {:thread=>70115975803940, :consumed=>14, :traps=>0}
    |   {:counters=>{"runs"=>2, "msgs"=>26, "omsgs"=>0}, :nodes=>2, :size=>500}
    \--- .

The execution run ends as bob received a "tas" message. The execution has been saved to the database.

I then return the bob task to the execution with a "ret" message. The scheduler notices the message, instatiates an executor, which removes the bob task node and sends a "rec" message to the sequence 0. The sequence is supposed to send an execute message to its next child, but there are no more children, it's supposed to send a "rec" message to its parent node then. There are no parent node, the sequence being the root node. The execution ends.

    /--- run starts Flor::UnitExecutor 70115974805760 shell-cli-20171021.0830.mechulorucho
    |   {:thread=>70115974734660}
    |   {:counters=>{"runs"=>2, "msgs"=>26, "omsgs"=>0}, :nodes=>2, :size=>500}
   ucho 0_1 ret [task L4] hp:task m27s_ f.ret "stage b"
   ucho 0_1 rec [task L4] hp:task m28s_r3>3 f.ret "stage b"
   ucho 0 rec [sequence L2] hp:sequence m29s28r3>3 from 0_1 f.ret "stage b" vars:var0,_path,root,name
   ucho rec m30s29r3>3 from 0 f.ret "stage b"
   ucho ter m31s30r3>3 from 0 f.ret "stage b"
    |   run ends Flor::Logger 70115974783520 shell-cli-20171021.0830.mechulorucho
    |   {:started=>"2017-10-21T08:42:46.923942Z", :took=>0.414913}
    |   {:thread=>70115974734660, :consumed=>5, :traps=>0}
    |   {:counters=>{"runs"=>3, "msgs"=>31, "omsgs"=>0}, :nodes=>1, :size=>457}
    \--- .

Benjamin, further questions via Twitter, the flor mailing list or the flor chat are welcome. I'll try my best to write a response blog post if necessary.

Thanks for asking.