Mule 4 OpenTelemetry Module

From a purist standpoint (i.e., based on the Gartner definition), who knows - maybe never. However, from a practical perspective the "messy" composable enterprise is already here, has been for a while and it’s quickly getting more "pure" over time.

For example,

So as you can see, the composable enterprise already exists and will, rapidly, become more composable over time, especially, with the support of companies like MuleSoft, products like the Anypoint Platform and methodologies like API-Led Connectivity.

The obtainment of true observability relies upon 3 core pillars.

The notion of observability is very important to IT organizations because when a business transaction fails or performs poorly within their application network, the team needs the ability to quickly triage and remediate the root cause before there is any significant impact on revenue.

Many IT organizations have and continue to rely upon commercial Application Performance Monitoring (APM) tools (e.g., AppDynamics, Dynatrace, New Relic, CA APM, …​) to help them in this regard. While useful, these commercial tools have struggled in the past to provide complete visibility into the overall distributed trace as they deploy vendor specific agents to collect and forward their telemetry.

I state "struggled in the past" because many APM vendors are now starting to embrace and support open source projects like OpenTelemetry for vendor-agnostic instrumentation agent implementations and standards such as W3C Trace Context for context propagation to help them fill in the "holes". See Vendor Support for OpenTelemetry - 2021 below.

Hopefully, the answer is obvious but as enterprise applications become more and more composable, that is, as enterprises move towards embracing composability as an architectural pattern, the need for observability becomes greater; however, the capacity for implementing observability becomes harder unless there is comprehensive observability strategy and solution in place.

Being a CNCF supported project, it’s no surprise the architecture of OpenTelemetry is cloud friendly - which also implies that it is friendly to all distributed environments. While there are various aspects to the overall OpenTelemetry framework (e.g., API, SDK, Signals, Packages, Propagators, Exporters, etc.), the functional architecture is relatively simple with regard to client-side implementations as seen in the diagram below.

On the client side (e.g., the Mule application), there are really only two OpenTelemetry components which are used and one is optional:

Below is a brief description of these client-side components.

As shown in the graphic below, a 2021 GigaOm study concluded that top tier cloud providers are moving to embrace OpenTelemetry quickly and observability vendors are likewise offering integration with OpenTelemetry tools, albeit, at various levels. However, it should be of no surprise the Gartner "visionaries" are offering the greatest level of support.

The GigaOm study also reveals full adoption of the OpenTelemetry standards can yield significant benefits around instrumentation, as customers can deploy drop-in instrumentation regardless of the platform. Furthermore, portability becomes achievable as well, improving both cost savings and efficiency.

The WC3 Trace Context specification defines a universally agreed-upon format for the exchange of trace context propagation data - referred to as trace context. Trace context solves the problems typically associated with distributed tracing by:

Trace context is split into two individual propagation fields supporting interoperability and vendor-specific extensibility:

Describes the position of the incoming request in its trace graph in a portable, fixed-length format. Every tracing tool MUST properly set traceparent even when it only relies on vendor-specific information in tracestate

Extends traceparent with vendor-specific data represented by a set of name/value pairs. Storing information in tracestate is optional.

Tracing tools can provide two levels of compliant behavior interacting with trace context:

The traceparent header represents the incoming request in a tracing system in a common format, understood by all vendors.

The header has 4 constituent parts, where each part is separated by a -:

Since the tracestate header is optional, it will not be discussed any further in this document. See WC3: Tracestate Header for additional details on the header.

In Mule 4, extending the product is done by developing custom extensions via a MuleSoft furnished Java SDK. The comprehensive framework allows external developers to build add-on functionality in the same manner as Mule engineers build Mule supplied components and connectors. While we won’t get into the details of the framework or how to develop a custom extension, the graphic below depicts the basic structure of an extension based on the Module Model.

Mule provides an internal notification mechanism that can be used to access changes which occur on the Mule Server, such as adding a flow component, the start or end of a message processor, a failing authorization request and many other changes. These notifications can be subscribed to by "listeners" either programmatically or by using the <notifications> element in a Mule configuration file.

The agent takes advantage of the notification framework and in particular relies upon these two notification interfaces:

The Mule 4 OpenTelemetry Module leverages the OpenTelemetry Java implementation to generate, batch and export trace data to any OpenTelemetry compliant Collector. Specifically, the agent builds on top of the opentelemetry-java package for manual instrumentation of Mule applications. By taking full advantage of the OTel Java implementation, the Mule extension becomes completely stand-alone and does not require any additional OpenTelemetry components to be a participant in a distributed trace.

The architecture of the Mule 4 OpenTelemetry Module is relatively straight forward. As depicted in the diagram below, the agent is comprised of code which listens for notification events from the Mule runtime. During the processing of the notification, the agent generates metadata about the notification and sends that data to the OpenTelemetry SDK via the OpenTelemetry API - shown as Trace Data in the diagram. The OpenTelemetry SDK continues to gather the extension generated trace data until all processing is complete. At that point, the OpenTelemetry SDK exports the trace data using the configured Exporter, e.g. the OpenTelemetry wire Protocol (OTLP) Exporter, to an OpenTelemetry Collector.

Extension uses the OpenTelemetry SDK’s Resource Provider SPI to gather data of common resources such as Host, OS, Processes of the Mule runtime Server and host. SDK supported all common resources providers are configured by this module.

See Disabling Common Resource Providers if any resource provider must be disabled.

Extension is aware of CloudHub Reserved Properties and automatically adds some of that data into trace data.

Extension contains all dependencies needed to send supported signals to an OpenTelemetry Collector endpoint.

When troubleshooting generated signal data, sending it to logs may be useful. Extension supports a simple logging exporter that can send signal data to application’s log file.

Example span entry from log file -

This generic exporter allows to configure any other signal exporters supported by sdk-extensions/autoconfigure#exporters.

Following example shows possible configuration for sending traces to Zipkin.

The required Zipkin exporter dependencies must be configured as an Additional Plugin Dependencies for Mule Maven Plugin.

For non-logging exporters, Tracing SDK uses Batch Span Processor. Global Configuration allows to customize Batch span processor settings -

By default, every span is recorded. In a high transaction environment, this can become noisy or needing high storage requirements for backend APM. In such cases, it is possible to reduce the span recorded and sent to the APM. This can help reduce the network traffic as well as data sent to the backend. Although, it comes at a cost of not collecting all traces, which maybe acceptable in certain use cases.

The Sampler configures weather spans will be recorded when they are started by the module. Unrecorded spans are skipped from exporting to backend APM.

Module config doesn’t have any elements to set the sampler configuration, but it can be applied using OpenTelemetry’s system properties.

See Sampler documentation for more details.

For example, to set the TraceId Ratio based sampler, you can add following two properties -

By default, this module will create trace spans for following mule components -

More verbose span generation can be configured. See setting Trace Levels below.

In addition to all the trace attributes captured by the module, it is possible to add custom tags to the current trace using an operation opentelemetry:add-custom-tags.

These could be any business data that you may want to capture as a part of your telemetry data. For example, an order number for an order processing transaction.

You can also use dataweave to set the tags.

Some APMs may require additional tags on spans for the correct display of traces. For example, Splunk APM can use peer.service on http request spans when creating inferred services in service maps.

The module may not be capturing those tags out of the box but there is a way to add additional tags to the spans of components that use global configuration elements. Some examples would be http:listener using http:listener-config, db:insert using db:config.

For such cases, the module recognizes system properties defined with property names following the pattern {global_config_element_name}.otel.{tag_name} and adds {tag_name}:{property_value} as tags to spans generated for all components using {global_config_element_name} named global element.

Few things to consider when using Global Tags:

In case of Splunk, peer.service attribute should have the name of the remote http system being invoked. Consider following mule requester example -

To add a tag peer.service=my_remote_api to http:request 's span, you can set following system property on mule runtime -

This module supports context propagation in W3C Trace Context and W3C Baggage Context formats.

Once you have configured the module in your application, there may be a need to remove or temporarily turn it off.

To permanently remove the tracing -

To temporarily disable the tracing without any code changes -

To add trace ID and span ID, you may add a mule tracing module to target the Mule application.

In the first APIKit flow, add tracing module operations -

When logging with Log4J, JSONLayout can be used to structure the log records with additional attributes.

To send traces to Elastic, at the minimum you need following information -

The below is an example of a working module configuration for Elastic Observability. We will look at some elastic specific configuration values.

Here is how a generated trace would look like in Elastic -

Trace attributes are available under Metadata tab for each trace transaction.

Below is a description of the demo scenario used to generate distributed trace information from 2 Mule applications and have it render in a Dynatrace backend.

Below is an implementation of the demo architecture described above. At a high-level, Mule_App_1 receives the initial request from the external client, performs various functions including making a request to an external application, Mule_App_2, and calling a secondary flow within Mule_App_1 before returning a response to the calling client application.

Below are several screenshots from a Dynatrace Distributed Traces Dashboard to provide examples regarding the type of output generated by the Mule 4 OpenTelemetry Module and visualized by observability backend.

Splunk provides the Splunk Distribution of OpenTelemetry collector to send OpenTelemetry based trace data to APM.

Once you have the collector installed and configured, you can use the module’s OTLP Exporter to send trace data to Splunk Observability Cloud.

Here is how a trace would look in Splunk APM -

Trace attributes are available when you expand individual span.

Depending on how global configurations are written they can be Static or Dynamic.

For Static configurations, a single instance is created and re-used each time it is needed.

For Dynamic configurations, Each time an operation that points to a dynamic configuration is executed, all the parameters in the configuration are evaluated. A configuration instance is created for each unique set of resolved parameters. In that way, each time the same values are resolved, the same instance is returned, but for each different set of values, a different instance is created.

This lifecycle difference can make a huge difference on the configuration objects created. Every request will have a different traceparent value. Thus, when using Dynamic configurations, a new instance will be created for each request.

Following result demonstrates the configuration Object creations for a load of 20 concurrent users for a 1-minute time period -

Observations:

Recommendations:

For higher load applications, static configurations i.e. propagation from operations instead of global configuration should be preferred.

Static Configuration Object Creations

Dynamic Configuration Object Creations