The Process Foundry

Welcome to The Process Foundry (TPF), a business process model and notation (BPMN) based microservice framework.

Be warned, like all frameworks it will suck. Read ahead to see if it can be made to suck less for you.

The goal of this projects is to create a backbone to implement inter-process communications between disparate processes on so the developer can focus on their code rather than digging into every niggling decision that goes into a microservice.

It is designed to be composable, but with sane defaults that the dev will never even know about unless they go looking.

Ideas to mull over

THINK: Is it worth creating a book preprocessor to scan the code for ToDo listing to include RFC/ToDos with links to project management in the book? THINK: Does Github's project management offer enough to use (even temporarily?)

ToDo

  • Add project management tool to contain bugs/feature requests
  • CODE REVIEW for documentation
  • Refactor into multiple projects and squash into a proper initial commit
  • Design workflows
    • Review Luigi/Airflow docs for ideas
  • Implement watcher.sh as TPF workflow (dog-fooding)
    • Create iNotify Application
    • Emit file touched event
    • Does workflow actually do anything after triggering, or does it just configure pub/sub and disappear?
    • Review choreography vs orchestration
  • Add ELK stack as a system in order to enable logging
  • Create TPF GUI
    • Research compiling WebAssembly container/cache and wrapping it in JavaScript
    • Is Kibana useful here, since it claims to be a dashboard/insight manager?

Core

This is the bare minimum needed to run the process foundry. It consists of the action/event aggregator, message bus, and shell containers. Most everything else is optional or considered a tool that can be used everywhere.

TODO: Add list, links and blurbs to the architecture.

Architecture

This is the top level design of The Process Foundry. Read the sub-headings to learn more about the individual components.

Application

An application is an interface which defines a set of functions to convert TPF actions into native commands, monitor state, and sends out messages describing any changes in state.

At its core, an application is something that executes actions and emits events. It has no context of anything outside the scope of its functionality. It can be accessed by one or more containers, depending on the context.

A primary example of this is the user shell

TODO: Add component/function design png

Container

A container is an introspective object that holds a set of applications and other containers (hooray for recursive definitions). It is used to organize its children and route events/actions to the next step in both directions, up to the message bus/router, and down to the "bare metal". This is what will enable app discovery.

TODO: Add PNG of docker-compose example or should this wait until workflows?:

  • docker-compose -> docker-container -> shell -> pg_basebackup
  • iNotify -> docker-compose -> docker-container -> shell

Workflow

Status: Thinking through

A workflow is a set of actions to run based on a specific change in state. It should be choreography rather than orchestration. The bulk of this is going to be to adding routing to make sure events get to the next proper step. It should be started with a single trigger and using routing to follow to subsequent events instead of having a process listener waiting for result before sending the next step. I'm not sure this is possible, but I'm damn well going to try.

TODO: Examine Luigi/Airflow for how they do things TODO: Workflow design doc TODO: Workflow example

Data Flow

The dataflow in this system is designed to be unidirectional. An action will will generate an ack event with the message id. Results are pushed onto the router as multiple applications may care about the result even though, only a single action was triggered. This simplifies handling state changes and maintaining a cache.

Web pages should also be treated as an equal container, receiving events and updates for realtime data syncing. The idea is to compile a WebAssembly module and wrap it in JS so web users will be treated as equal to onsite containers/applications. This hopefully keep the codebase with bug fixes up to date without having to manage multiple programming languages.

I'm basing this off the idea of Flux

TODO: Add example data flow for the watcher/rebuilder workflow

API

Since this is a microservice framework, the bus will be exposed and secured by OAuth2 and have a queryable JSON schema for all available functions. The goal is to have 3rd party apps able to act as first class citizens.

GUI

status: Concept/ToDo

THINK: Kibana as a base for insight?

This is going to be able to give insight/management into all the systems known by TPF. First get the core backbone running before even considering what we would want to see.

TODO: Workflow management (editing, running) TODO: Role/ACL management

Tools

Applications

The application is an interface used for wrapping externally code. Here are some examples and how they are used.

TODO: Core::Shell (Bash, Zsh, Dash) TODO: Add QuickBooks API interface

Systems

This is a set of applications that are run together to perform a particular function. These are going to be predefined configurations for quick and easy deployment and management. In addition, they should add on pre-designed workflows for common usages.

ELK Stack

That's Elasticsearch, Kibana, Beats, and Logstash (also known as the ELK Stack). Reliably and securely take data from any source, in any format, then search, analyze, and visualize it in real time.

This is going to be the default lagging mechanism for The Process Foundry, so I'm including it as since we can install the whole thing in a Docker Compose/MiniKube instance.

To see more about it: Elastic Stack

Taiga

Fishhead Labs

My old company, still running under my sister. It is a full service veterinary pathology consulting firm. The company's internal management website is built upon TPF (such as it is).

Workflows

Continuous Backup

Rebuilder

Invoicing

Glossary

  • Container
  • Application
  • RBAC (Role Based Access Control)
  • ACL (Access Control List)
  • Encryption
  • Message Queue
  • Factory
  • OAuth2
  • ELK Stack
  • Registry
  • Router
  • Wrapi
  • Long Poller
  • System This is a set of applications that are run together to perform a particular function. A good example is the ELK Stack.