Skip to main content
Version: Self Hosted Lite

Custom Parser

What is Custom Parser?​

SnappyFlow's custom parser feature enables the collection and monitoring of logs in custom format. It proves useful for handling distinct data formats or specialized parsing needs in specific scenarios.

The SnappyFlow parser plugin is built using Fluent Bit constructs and SnappyFlow is supporting the representation of Fluent Bit components in YAML. To learn more about the representations of core components, please refer to the link provided below.

Fluentbit Component Representation in SnappyFlow

Create a Custom Parser​

Prerequisites​

  • Parser plugin should emit records with time field. The time representation should be associated with Epoch Millisecond value.

  • If the data is destined to log index, set type=logs and add a message field. The message field should not be empty.

Parser Plugin Format​

To create a custom parser, utilize the format specified below.

<-pluginName->
    documentType: <-docType->
    type: <-metrics/logs->
    pluginDisplayName: <-displayNameInUI->
    pluginGroupName: <-groupNameRenderedinUI->
    inputs:
    - List of Snappyflow Represented Input components
    parsers:
    - List of Snappyflow Represented Parser components
    filters:
    - List of Snappyflow Represented Filter components

Lua Filter Function​

Create a Lua filter Function as shown below:

function <-function-name-> (tag, timestamp, record)
// business logic to modify record table
 return 1, timestamp, record
end

sfAgent Plugin Configuration​

Utilize the format specified below to configure a plugin for logs.

logging:
  plugins:
    - name: <-pluginName-> #give the Plugin name in the parser plugin format
      enabled: true
      config:
        log_path: log path seperated by comma

Configure the sfAgent​

  1. Go to the Application tab in SnappyFlow and navigate to your Project > Application.

  1. Click the application, it will take you to the inventory page.

  2. Select the Configuration tab and click the Download configurations option to download the existing configuration.

  3. Downloaded collection will contain the below given files with pre-configured data.

    • config.yaml file
    • custom_logging_plugins.yaml file
    • custom_script.lua file

  4. Updated the pre-configured files a give below:

    • Add sfagent Plugin Configuration to the config.yaml file.
    • Add Parser Plugin Format to the custom_logging_plugins.yaml file.
    • Add Lua Filter Function to the custom_scripts.lua file.
note

Do not modify any existing contents. Add the parser configuration to the existing data.

  1. To upload the updated agent configuration files, go to the Application tab in SnappyFlow and navigate to your Project.

  2. Click the ... Project menu icon.


  3. Select the Agent Configuration option.

​

​

  1. In the Agent Configuration window, click the + Add New Bundle button.

  1. Use the Drag and drop or Browse option the add the updated agent configuration files.
  2. Give a Name to the bundle and add a description.
  3. Click the Upload button.

Apply the Custom Parser​

Follow the below steps to apply the custom parser configuration to an application.

  1. Go to the Application tab in SnappyFlow and navigate to your Project > Application.

  2. Click the application, it will take you to the inventory page.

  3. Go to the Configurations tab.

  4. Click the Apply Configuration option.

  5. Click the Apply option of the new bundle.

  6. In the Configuration Operation pop-up window, select the Plugin Config Update option and click Ok.