Business Events Capture - Logs

Organizations in today’s data-driven world often struggle with fragmented data sources that hinder comprehensive business insights. With Dynatrace OpenPipeline, you can ingest logs from any system and extract relevant business data to get a cohesive end-to-end view of your business processes.

This approach is useful if logs contain business-relevant information or no other ingest path for business events is available.

Log to Business events documentation

This lab will utilize logs lines ingested by the OneAgent. The log lines will be converted to business events for the Fraud Check step for the Astroshop Order to Shipped business process.

• Use OpenPipeline to convert incoming logs to business events
• Validate Business Event data with a Notebook

Fraud Check

In this section of the lab we will use OpenPipeline to convert incoming logs to business events for the Fraud Detection step of the Order to Shipped business process.

Logs - Fraud Check

Using a Notebook execute the below DQL query which retrieves the log lines that will be converted into business events for the Fraud Check step of the Order to Shipped business process.

DQL:

fetch logs
| filter k8s.container.name == "frauddetectionservice"
| filter matchesPhrase(content, "Consumed record with orderId:")
| filter matchesPhrase(content, "orderId")
| fields timestamp, content
| sort timestamp desc

Result:

Logs - Fraud Check orderID Extraction

Using a Notebook execute the below DQL query which retrieves the log lines that will be converted into business events for the Fraud Check step of the Order to Shipped business process. We will parse the logs lines to extract the orderId.

DQL:

fetch logs
| filter k8s.container.name == "frauddetectionservice"
| filter matchesPhrase(content, "Consumed record with orderId:")
| parse content, """DATA 'Consumed record with orderId: ' ld:orderId ','"""
| fields timestamp, orderId, content
| sort timestamp desc

Result:

Business Event - OpenPipeline - Fraud Check

Launch the OpenPipeline app.

Select Logs in the tree.

Select the Pipelines tab.

Select the + Pipeline button.

Business Event - OpenPipeline Processing - Fraud Check

Name your OpenPipeline rule using the following name:

Astroshop_Fraud_Check

Select the Processing tab

Select the + Processor button (left side of the screen), then select DQL

Fill out the fields with the following data:

Name:

FraudCheckBizEventFromLog

Matching condition:

k8s.container.name == "frauddetectionservice" AND matchesPhrase(content, "Consumed record with orderId:")

DQL processor definition:

parse content, """DATA 'Consumed record with orderId: ' ld:orderId ','"""
| fieldsAdd add1second = toTimestamp(toLong(timestamp) + 1000000000)
| fieldsAdd timestamp = add1second

In the sample data section use the following:

{
      "timestamp": "2025-03-19T13:41:16.000000000-04:00",
      "content": "2025-03-19 17:41:16 - frauddetectionservice - Consumed record with orderId: 5ba0f19d-04e9-11f0-895b-0e326b3139ca, and updated total count to: 1126 trace_id=6cd0e9c60fd10799c66eb344f916d7e6 span_id=3173a5d07a507a1d trace_flags=01 ",
      "k8s.container.name": "frauddetectionservice"
}

Select the Run sample data button and Preview the results.

Validate the orderId field exists in the results.

Business Event - OpenPipeline Data extraction - Fraud Check

Select the Data extraction tab.

Select the + Processor button (left side of the screen), then select Business event.

Fill out the fields with the following data:

Name:

orderId

Matching condition:

k8s.container.name == "frauddetectionservice" AND matchesPhrase(content, "Consumed record with orderId:") and isNotNull(orderId)

Event type:

Select the Static string radio button and use the following:

astroshop.fraudcheck.success

Event provider:

Select the Static string radio button and use the following:

astroshop

Field extraction:

Select the Fields to extract radio button. We will add 3 fields to extract (keep). The fields need to be added independently.

Click the Add button for:

timestamp

Click the Add button for:

orderId

Click the Add button for:

trace_id

Select the Save Button at the top right of the screen.

Business Event - OpenPipeline Dynamic Route - Fraud Check

Now we need to create a Dynamic route for the Astroshop_Fraud_Check pipeline.

Dynamic routes give you the option to route your ingested data into a pipeline with a matching condition.

In the OpenPipeline App, select Logs in the tree.

Select the Dynamic routing tab.

Select + Dynamic route button.

Configure the Dynamic route with the following:

Name:

Astroshop_Fraud_Check

Matching condition:

k8s.container.name == "frauddetectionservice" AND matchesPhrase(content, "Consumed record with orderId:")

Pipeline:

In the dropdown list select the following

Astroshop_Fraud_Check

Select Add button.

Select Save button.

A warning icon with this message will appear "Do you want to save your changes to this table?"

Select the Save button.

Conclusion

We have completed the Business Event capture using log lines and OpenPipeline for Fraud Check step of the Order to Shipped business process. The next section will validate the data.

Data Validation

In this section of the lab we will validate the data for the Fraud Check step of the Order to Shipped business process.

Query Business Events in Dynatrace

Using a Notebook execute the below DQL query which retrieves the business events for astroshop.fraudcheck.success step. Note, this may take 3-5 minutes for new data to appear.

DQL:

fetch bizevents
| filter event.provider == "astroshop" and event.type == "astroshop.fraudcheck.success"
| fields timestamp, event.provider, event.type, orderId, trace_id
| sort timestamp desc

Result:

Conclusion

We have completed data validation for the for the Fraud Check step of the Order to Shipped business process.

Continue

In the next section, we'll capture Business Events with the Dynatrace API.

• Continue to Business Event Capture with API