flows: Resilient and Declarative E2E Workflows and automation

At Lab34, we often build systems that handle complex, multi-step processes. Think about a typical user registration, an e-commerce checkout process, or an IoT device responding to a command. Each process involves multiple services - from APIs and database transactions to asynchronous protocols or slow internet connections. Each step has its own peculiarities. Each step depends on the last. Any of them could fail.

Testing and managing these processes end-to-end can be challenging. You might write complex integration tests, but these often become brittle, hard to understand what actions they really perform, and could endup giving little or no pace for (fuzzing)[https://en.wikipedia.org/wiki/Fuzzing].

And most anoyingly for developers, in some circunstances do not pass on local environments or offer no help for validating OK/NOK scenarios locally.

We wanted a better way—a solution that would allow us to define these processes in a clear, resilient, and declarative manner, without needing to write complex testing code. That’s why we created and are now open-sourcing lab34-flows, a powerful orchestrator for defining and running workflows using simple YAML files.

What is lab34-flows?

lab34-flows is an orchestration tool that executes multi-step workflows defined in YAML. Instead of writing code to glue services together, you describe the sequence of operations in a nearly-every-human-readable format. This makes it an ideal tool for End-to-End (E2E) testing, process automation, and defining complex business logic.

While the flows are defined in YAML, the core engine and the steps themselves are built with JavaScript (nodejs), giving you the underlying benefits of robustness without exposing that complexity to the user writing the workflow.

The Architecture: Understanding the Key Components

To understand how lab34-flows works, it helps to break it down into three core concepts: the tool, the context, and the credentials.

The Tool (lab34-flows): This is the core engine (a cli command) you’ve just read about. It’s the orchestrator that reads and executes the YAML workflow files. Its main job is to parse the steps, call the appropriate application methods, manage the flow of data, and run the defined tests.

The Organization’s “Context”: This is a shared folder or repository where developers define the integrations. It acts as a bridge between the simple YAML files and your actual services. Here, developers write the code that implements what an application and method actually do. For example, the stripe_gateway application and its tokenize_card method are defined in the context. This allows anyone in the organization—including QA, DevOps, or product managers—to write and review test scenarios in simple YAML without needing to know the underlying implementation details.

The “Credentials” (.env files): These are standard .env files where environment-specific configurations like URLs, database credentials, and API keys are stored. This practice keeps sensitive information out of the YAML files and the context repository. By simply swapping out the .env file, you can run the exact same workflow against a local mock, a staging environment, or even production, ensuring consistency across your entire pipeline.

The Core Concepts: A YAML-First Approach

Workflows in lab34-flows are defined with a straightforward YAML structure:

• title & description: High-level information about what the flow or test case does.
• steps: An ordered list of actions to be executed. Each step in the list is a self-contained operation.
• application: The name of the service or component you want to interact with (e.g., lrs, iot_locker, stripe_gateway).
  • method: The specific action or function to call on that application (e.g., reservation, login).
  • parameters: The data required by the method, such as request bodies or IDs.
  • test: This are the rules that must match (if any) to consider an step successful.

Here is an example of what a test case could look like

title: Check product stock.
description: Some optional description.

steps:
  - application: product_catalog
    method: api_create_product
    # Parameters are often optional.
    # If none provided, they are randomly generated.
    parameters:
      body:
        name: R/C Car
        
  - application: product_catalog
    method: database_searh_ean
    # no parameters are needed, since flows can keep "memory" of other steps'
    # outcomes
    # parameters:
    #   body:
    #     ean_code: "{{ steps.product_catalog.database_searh_ean.result.body.ean }}"
    test:
      body:
        stock_amount: 0       # validate that initial stock is present 
      retry:                  # This could be an asynchronous process, so:
        delay: 5000           #   perform the check every 5 seconds 
        times: 10             #   up to 10 times

As you can see, the flow is defined declaratively (as much as you need). There’s no code, just a clear description of the applications involved and the actions to perform. lab34-flows takes this file and executes the steps in order, handling the state and data flow between them.

Defining Tests within Flows

Beyond simple execution, lab34-flows is a powerful testing tool. You can define assertions directly within a step using the test key (apple-to-apple comparisons, but also evals). This allows you to validate the outcome of each action, turning your workflow into a robust E2E test case.

Your tests can include:

• Status Code Validation: Ensure an API call returns the expected HTTP status.
• Headers: to ensure certain data is present in HTTP response headers.
• Response Body Assertions: Check the content of a response, from simple value matching to complex expressions.
• Retry Logic: Automatically retry a step for a configurable number of times, which is essential for testing asynchronous processes where results aren’t immediate.

A More Practical Example: Online Payment Processing

Let’s look at a more complex process: handling a payment on an e-commerce platform. With lab34-flows, the entire E2E test, including validations and handling of asynchronous jobs, can be defined in YAML.

title: Process New Customer Payment
description: This flow tests the entire payment process, from user signup to transaction reporting.

steps:
  - application: user_service
    method: signup
    description: Creates a user record if one doesn't exist
    parameters:
      body:
        userEmail: "customer@example.com"
        name: "Jane Doe"
    test:
      status: [201] # Assert that the user was created successfully

  - application: stripe_gateway
    method: tokenize_card
    description: Securely gets a token for card details
    parameters:
      body:
        cardNumber: "4242..."
        # ... other card details

  - application: payment_db
    method: create_record
    description: Creates internal DB records for the transaction
    parameters:
      body:
        amount: 1000
        currency: "USD"
        # The flow can pass the user ID and stripe token from previous steps

  - application: reporting_service
    method: trigger_report_generation
    description: Kicks off an asynchronous report generation job

  - application: reporting_service
    method: get_report_by_transaction
    description: Checks that the report has been successfully generated
    test:
      body:
        status: "COMPLETED"
      retry:
        times: 10
        delay: 5000 # Poll for up to 50 seconds for the async report to complete

This YAML file provides a clean, expressive, and robust definition of the entire transaction. If the user_service returns a status other than 201, the flow halts immediately. It also gracefully handles the delay in the reporting service by polling until the job is complete.

Local Development and End-to-End Testing

One of the biggest challenges in developing complex systems is dealing with external dependencies. How do you run an E2E test of your payment flow without making real API calls to Stripe or your bank? Or when you are behind a corporate firewall: How do you test against other application’s API?

lab34-flows is designed to solve this. Because each application is a modular component defined in the Context, you can configure the flow runner to substitute a real service with a local mock. For example, in a local or CI environment, you can configure lab34-flows so that the stripe_gateway application points to a mock server you control, which can simulate success or failure scenarios dinamically.

This help you test an scenario where credit card “1234-1234-1234-1234” is expected to work, but “1111-1111-1111-1111” is expected to fail. From the comfort on your localhost.

This allows your developers and QA teams to execute complete E2E flows on their local machines, even without access to all production services, dramatically speeding up development and testing cycles.

Get Started with lab34-flows

We built lab34-flows to solve our own testing and orchestration challenges, and we believe it can help others build more reliable and maintainable applications.

The project is open-source and available on GitHub. We encourage you to try it out in your next project.

• GitHub Repository: https://github.com/lab34-es/lab34-flows

Dive into the documentation, explore the examples, and let us know what you think. We welcome contributions, feedback, and ideas from the community!

Key Features at a Glance

• Declarative YAML Workflows: Define complex tests and processes in a simple, human-readable format. No coding required for test authors.
• Shared Application Context: A central, version-controlled repository for defining service integrations, enabling collaboration between developers and testers.
• Environment-based Credentials: Securely manage API keys, URLs, and other secrets for different environments (local, staging, prod) using .env files.
• Built-in Assertions and Retries: Validate outcomes directly in your YAML with status, body, and retry checks, perfect for asynchronous workflows.
• Automatic Error Handling: Fail-fast error handling stops the flow on any exception, preventing inconsistent states and providing clear reports.
• Enhanced Testability: Easily mock any application to enable robust and fast E2E testing in local and CI environments.
• Composable and Reusable Steps: Build flows from a library of pre-defined, robust steps that can be reused across your application.
• Robust Core Engine: Built with TypeScript for a reliable and type-safe execution environment under the hood.