Configure Flow

This page explains how to set up a flow to filter, shape, and route data from an Axiom dataset to a destination.

Flow is currently in private preview. To try it out, sign up for a free preview.

A flow is a way to filter, shape, and route data from an Axiom dataset to a destination that you choose. This page explains how to set up a flow.

Prerequisites

• Create an Axiom account.
• Create a dataset in Axiom where you send your data.

• Set up a destination. For more information, see Destinations.

Set up a flow configuration

To set up a flow configuration:

1. Click the Flows tab. Axiom displays the list of flow configurations you have created.

2. In the top right, click New configuration.

3. In the Source section, specify the source dataset and the transformation in an APL query. For example, the following APL query selects events from a cloudflare-logpush dataset and reduces them by removing a set of fields, before enriching with a new field.

   ['cloudflare-logpush']
   | where QueryName == "app.axiom.co."
   // Reduce events by dropping unimportant field
   | project-away ['@app']*
   // Enrich events with additional context
   | extend ['@origin'] = "_axiom"
   

   If you only specify the name of the dataset in the query, Axiom routes all events to the destination.

4. Click Preview to check whether the query you specified transforms your data as desired. The Input event section displays the original data stored in Axiom. The Output event section displays the transformed data that Axiom sends to the destination. The original data in the Axiom dataset isn’t affected by the transformation.

5. In the Destination section, click Add a destination, and then select an existing destination where you want to route data or click Create new destination.

6. In the top right, click Create.

After creating a flow configuration, create a flow by selecting one of the following:

• Continuous flow
• One-time flow

Create continuous flow

Continuous flows are continuously running operations that process your incoming data and route the outputs to a destination in real-time.

1. Click the Flows tab. Axiom displays the list of flow configurations you have created. Select the flow configuration that you want to use for creating a continuous flow.
2. In the top right, click Create flow and select Continuous flow.
3. Click Create flow.

As a result, Axiom starts running the query on all incoming data and routes the results of the query to the destination.

Create one-time flow

One-time flows are one-off operations that process past data for a specific time range and route the output to a destination.

1. Click the Flows tab. Axiom displays the list of flow configurations you have created. Select the flow configuration that you want to use for creating a one-time flow.
2. In the top right, click Create flow and select One-time flow.
3. Specify the time range for events you want to process.
4. Click Create flow.

As a result, Axiom runs the query on the source data for the specified time range and routes the results of the query to the destination.

Delivery rate of continuous flows

The delivery rate of a continuous flow currently depends on the rate at which you ingest data into the source dataset. For example, if you set up a continuous flow and ingest data at a rate of 1TB/day to the source dataset, Axiom processes events within a few seconds after ingest. If you ingest data to the source at a rate of 10GB/day, it can take several minutes for events to arrive. There is currently no maximum wait time for events to be processed by a continuous flow. As Flow progresses through the preview stage, Axiom will establish and refine maximum wait times.