Monitoring APIs with BAT CLI

In today’s world, integrating our software with other systems isn’t just a luxury – it’s a necessity. Ensuring the proper quality of service becomes a more important and challenging task to do.

For example, let’s say we want to offer an API for our product. Through this API, clients can integrate their software with ours, whether they’re aiming to innovate their systems, make revenue, or just expand our mutual market presence. For a lasting integration, it’s crucial to ensure that our API consistently delivers the service outlined in the contract.

While today’s cloud infrastructure offers solutions for maintaining uninterrupted operation, making our applications up and running 24/7, one question remains: Are we confident that our product always behaves as expected?.

MuleSoft addresses this concern with Functional Monitoring, an integral feature of its Anypoint Platform.

What is API Functional Monitoring?

API Functional Monitoring in the Anypoint Platform is a feature that maintains the quality and dependability of public and private APIs within our application networks. It allows us to test APIs’ functional behavior and performance

Key functionalities include:

• Monitoring via the Anypoint Monitoring UI.
• Monitoring using the BAT command line interface (BAT CLI).
• Viewing and generating reports on monitor execution results.

Below is an image of the Functional Monitoring page with a selected monitor for reference:

How do we create API monitors?

We can create monitors through the UI by selecting from the array of options available. This process enables swift setup of tests for any of our deployed APIs. The following picture shows creating monitors via the Functional Monitoring UI:

When testing requires a finer degree of control or specific behaviors, scripting tests in the BAT framework and uploading them offers an ideal solution.

We can write our test scripts directly within the integrated code editor or create test suite projects using third-party IDEs such as IntelliJ or Visual Studio Code and later upload them to the Anypoint Platform. Developing our test suite as a standalone project allows us to implement version control and CI/CD pipelines, enhancing our development workflow.

Both methods empower us to establish tests that can be scheduled and executed at predefined intervals, ensuring comprehensive system monitoring. We have the option to choose between the simplicity of UI-based creation or the flexibility of BAT-based tests.

What exactly is BAT CLI, and how can you start using it?

BAT projects are not too complicated to develop. To start developing such a project, we first have to download and install the BAT CLI tool on our local machine. For details, please take a look at this document: BAT install.

Once installed and verified with the “bat –version” that everything is ok, we can scaffold the initial project structure with the “bat init” command or create everything manually. The basic BAT project structure looks like this:

.
├── bat.yaml
├── config
│   ├── dev-env.dwl
│   └── local-env.dwl
├── reports
│   ├── Report.html
│   └── Report.json
└── tests
    ├── QoS.dwl
    ├── Shippings.dwl
    └── ShippingsById.dwl

The “bat.yaml” file is the test suite’s main file; it must be placed at the root level and can’t be renamed. It is the place where we define the test files and which kind of reports to create. 

suite:
  name: "Shippings System BAT"

files:
  - file: tests/Shippings.dwl
  - file: tests/ShippingsById.dwl
  - file: tests/QoS.dwl

reporters:
  - type: HTML
    outFile: reports/Report.html
  - type: JSON
    outFile: reports/Report.json
  - type: Email
    options:
      EMAILS: shippings.system.bat@gmail.com

The files section is where we point to the test files. The sequence in which these test files appear in the “bat.yaml” file dictates their execution, ensuring that tests run in the specified order and maintaining the integrity of the testing process.

The reporters section allows us to configure different tools or third-party applications to report test results. In our scenario, it will generate an HTML page and a JSON file. It will also send an alert email if any of the tests fail.

Besides those, we can also integrate BAT with Slack, New Relic, PagerDuty, and Sumo Logic. This comprehensive reporting system ensures that we receive timely and detailed feedback on the status of our tests.

We can store variables in the files in the config folder. For example, suppose we’re testing an API deployed to two different environments. In each configuration file, we can place a variable named “URL” that holds the endpoint URL for the corresponding environment. When we want to run the BAT script, we can also instruct which configuration to use, for example: “bat –config=dev-env.”

The tests directory holds our actual test scripts, which will be executed against our API endpoints. The team writes test scripts in DataWeave 2.0 language, using the Behavior Driven Development (BDD) syntax.

Let’s say we have the Shippings System API deployed, and we aim to test its “/shippings” resource. According to the API specification, developers expect that this endpoint will consistently return data in the form of an array of JSON objects or an empty array if no records exist in the database.

We can implement several assertions to verify that our API functions as intended. These assertions ensure that the endpoint consistently returns the expected attributes, including the status code, MIME type, and the payload data structure.

%dw 2.0 
import * from bat::BDD
import * from bat::Assertions

fun getShippings(): Object = (
    GET `$(config.url)/shippings` with { 
allowUnsafeSSL: true
    }
)
---
describe `Test /shippings resource` in [
  it must "get all shippings" in [
      getShippings() assert [
          $.response.status                       mustEqual 200,
          $.response.statusText                   mustEqual "OK",
          $.response.mime                         mustEqual "application/json",
          (typeOf($.response.body) as String)     mustEqual "Array"
      ] 
  ]
]

BDD syntax gives us a lot of options for controlling our tests. We can use the predefined keywords to run a complete test suite or run just some tests selectively. If we get a failure in the middle of a test, we need to delete any created assets. For example, the user creates an asset, performs a validation, and then deletes it.

Typically, that validation fails, and since it breaks the test, the asset is not deleted, and our database starts accumulating test data. We can fix this simply by changing a “must” to a “should” so that execution can continue but not leave a failed asset.

%dw 2.0 
import * from bat::BDD
import * from bat::Assertions

var context = bat::Mutable::HashMap()

fun createShipping(): Object = (
    POST `$(config.url)/shippings` with {
        allowUnsafeSSL: true,
        body: { ... }
    } execute [
        context.set("shippingId", $.response.body.shippingId default 0)
    ]
)
---
describe `Test /shippings resource` in [
    it should "create new shipping" in [
        createShipping() assert [
            $.response.status                   mustEqual 201,
            $.response.statusText               mustEqual "Created",
            $.response.mime                     mustEqual "application/json",
            (typeOf($.response.body) as String) mustEqual "Object",
            $.response.body.status              mustEqual "Shipping successfully created",
            $.response.body.shippingId          mustEqual context.get("shippingId")
        ] execute [
            if (context.get("shippingId") != 0) (
              deleteShippingById(context.get("shippingId"))
            ) else (
           	 log("Shipping was not successfully created. There is nothing to delete...")
            )
        ] 
    ]
]

Also, we can execute tests in loops, which proves valuable when assessing functionalities like the Rate-Limiting policy within our API. This approach allows us to simulate various scenarios and stress-test the behavior of our system under different conditions.

%dw 2.0 
import * from bat::BDD
import * from bat::Assertions

var context = bat::Mutable::HashMap()

fun getShippings(): Object = (
    GET `$(config.url)/shippings` with { 
        allowUnsafeSSL: true
    } execute [
        context.set("response", $.response default "")
    ]
)
---
describe `Test Quality of Service policies` in [
    // Must return Bad Request, since API Quota Limit is 15 requests per minute.
    it must "reach quota limit" in [
        repeat(30) times [
            getShippings() 
        ] assert [
            context.get("response").status          mustEqual 429,
            context.get("response").statusText      mustEqual "Too Many Requests",
            context.get("response").mime            mustEqual "application/json",
            context.get("response").body.error      mustEqual "Quota has been exceeded"
        ]
    ]
]

When we want to set up BAT tests in the cloud, we also have the option to utilize the BAT CLI integration with the Anypoint Platform. We can upload and schedule tests using the CLI-xAPI by incorporating BAT CLI with the platform. The initial step involves logging into our Anypoint Platform account:

bat login --username="our-username-here" --password="our-password-here" --host="anypoint.mulesoft.com"

Once we log in, we must check available locations to choose where to run our tests:

bat location ls
organizationId: 276199ab-4d00-470e-8bb5-ddfaf2412be9
locations:
-
  id: 75c403a6-8054-43ec-b611-63b9efff820d
  name: us-east-1
  transport: LAMBDA
  organizationId: public
-
  id: 799baaeb-3c86-4798-b5b4-6d6954254229
  name: us-east-2
  transport: LAMBDA
  organizationId: public

To upload the test suite, we have options to specify the name, configuration file, and schedule time (defined using a cron expression). The system sets the default schedule time to 15 minutes if no specifications are provided. Additionally, indicate the location where we intend to upload our monitor:

bat schedule create --name="Shippings System BAT" --config=dev-env --cron="0 0 0/1 1/1 * ? *" --location=75c403a6-8054-43ec-b611-63b9efff820d

For more options with BATs, please refer to the official documentation: API Functional Monitoring with BAT CLI, especially the “BDD Test-Writing Syntax” and “BAT CLI Reference” pages.

You can find the code mentioned in this blog post in my public repository if you’re curious about it. There is a complete BAT project with tests, scheduled to regularly check the functionality of an API deployed on the Anypoint Platform:

• https://github.com/danijeldragicevic/shippings-system-bat

Feel free to dive into the test suite and explore the provided examples to better understand how to interact with the BAT.