Agent-based G2 Custom Monitors (Remote Script Executor)

Using G2 custom monitors, you can build and manage independent monitoring solutions without relying on the platform. Remote Script Executor executes shell, PowerShell, and Python scripts on Linux and Windows workstations.

Key features include:

• User-defined external arguments support the script through custom attributes.
• Monitor level script execution: Ability to pull more metrics in a single script execution.
• Ability to execute different types of scripts through custom script options apart from default options.
• Ability to use credentials attached to the device in the script using the dynamic macro support.

Constraint:

• Windows workstation monitoring using a Linux Gateway is not supported.

• If the end device language is other than English, ensure that the resulting JSON output from the script contains valid English characters, converting any invalid characters as necessary.

  Example:

  For the Monitor ‘Agent G2 - Microsoft AD Global Catalog Search Time’, the resulting output is:

      {'microsoft_AD_global_catalog_searchTime" :{"Components":
  
      {121.0.0.1":"13,64"}        
      }}
      

  In this example, we received ‘13,64’ instead of ‘13.64’ due to the end device language being Finnish. To convert this to English, we made the following change in the script:

  $formattedTime = "{0:N2}" -f $bindTimeMilliseconds.ToString("N2",[System.Globalization.CultureInfo]::InvariantCulture)

  After this change, we obtained the valid JSON output:

      {'microsoft_AD_global_catalog_searchTime" :{"Components":
  
      {121.0.0.1":"13.64"}
      }}"
      

Step 1: Create Metrics

Define metrics metadata that correlates with your script output and matches the monitor you define in the next step.

1. Select a client from the All Clients list.
2. Go to Setup > Monitoring > Metrics.
3. From METRICS, click + Add.
4. From Create Metric, enter the values for the fields described in the following tables and click Save.

Metric specification

See the Metric token reference for a list of the tokens that is used in the Subject and Description fields.

Step 2: Create a Monitor

A custom Remote Script Executor monitor is a collection of Remote Script Executor metrics. You can create a template based on the Remote Script Executor monitor.

Macros are used to pass dynamic arguments to scripts. Use the static and dynamic macros listed in the following tables to make native and custom attributes available to the monitor.

You must create metrics in Step 1: Create metrics before creating a monitor.

1. Select a client from the All Clients list.
2. Go to Setup > Monitoring > Monitors
3. Click + Create New.
4. On the CREATE A MONITOR page, enter the fields listed in the following table and click Save. The newly created monitor is added to the list of monitors.

Monitor specification

Static macros

Use static macros to override the resource values. See the Static macro reference for a list of static macros.

Dynamic macros

Below are the supported credential types:

[{
"uuid":"xB35mBgN354UGM3zQFZBQSaJ",
"type":"VMWARE",
"name":"sample-vmware",
"timeoutMs":10000,
"transportType":"HTTPS",
"appName":null,
"domain":null,
"username":"testuser",
"password":"xxxxxx",
"port":443
}]

[{
"uuid": "NwKGzg6qqSF29Sdy5hRJrAyH",
"type": "SSH",
"name": "sample-ssh",
"timeoutMs": 10000,
"transportType": "HTTP",
"appName": null,
"userName": "testuser",
"password": "xxxxxx",
"port": 22,
"privkey": null,
"passPhrasePasswd": null,
"privKeyFileName": null,
"sshAuthType": "PASSWORD"
    }]

The following are the remaining credential types:
SNMP, XEN, WINDOWS, JMX, HTTP, Database, CIM, NETAPP, NETAPPCLUSTER, HYPERFLEX, PURESTORAGE, FTP, CISCOUCS, EMCCLARIION, EMCVNX, EMCVNXE, EMCVMAX, IBM, HPEVA, REMOTE_CLI, TELNET, XTREMIO, VIPTELA, EMCVPLEX, EMCRPA, NUTANIX, HITACHIVSP, AZURESTACK, APPLICATION, and CITRIX_CVDA.

Script execution path for configuration parameters in Linux and Windows

The following runtime executables should be available in your Path variable for the corresponding operating system.

If your runtime is not set as an environment variable, provide the runtime absolute path in custom.script.execution.path. For example, if python is not set, set custom.script.execution.path to /usr/lib/python.

Step 3: Create a Template

A template is an instance of a monitor and is applied to devices.

1. Select a client from the All Clients list.
2. Go to Setup > Monitoring >Templates.
3. From Templates, click + Add.
4. From MONITOR TEMPLATE, provide details for the following parameters and click Save.

After adding the template, add component thresholds and component filters by editing metric values. For more information, see Add Filter, Component Filters and Define Threshold

Step 4: Assign a Template

1. From the left pane, click Infrastructure. The Infrastructure screen of the selected client is displayed.
2. From the Resources tab, select the required resource from the list of resources. Or, use the search option to find the resource.
3. Click the resource name to view details.
4. From the left pane, click Monitors.
5. From the Templates tab, click +Assign Templates.
6. From Apply Templates, select the templates.
   The selected Templates section displays the chosen templates.
7. Click Assign. The template gets assigned to the selected device.

After assigning the template to a resource for monitoring, click Get Latest Metric Values to view the latest metric information.

Step 5: View Graphs

The Agent monitors the system using the assigned templates and displays the results in a graphical format.

1. From the left pane, click Infrastructure.
2. From the Resources tab, select the required resource from the list of resources. Or, use the search option to find the resource.
3. Click the resource name to view details.
4. From the left pane, click Metrics. The Metrics page displays graphs generated by all monitoring templates assigned to a device.
5. Search with the template name to filter the graphs.

Scripts Folder and Permissions

By default, all the Remote Script Executor (RSE) custom monitor scripts are downloaded to a predefined location in your system. The system user (Windows) and Root/Non-Root User (Linux) hold all the required permissions (Read, Write, and Execute) in the downloaded folders.

The following table provides the default folder locations and users for Windows and Linux:

Script File Integrity Check

OpsRamp will verify for the custom script for file integrity and ensure that the file checksum remains the same before performing the monitor script execution. If the G2 based custom monitor script on the resource is changed, OpsRamp will redownload the original monitor script from the source for security reasons and continue the monitoring.

During this process, a critical alert about the “File checksum mismatch” will be generated on the resource, and users will be notified about the script modification on the resource.

Remote Script Executor example

Script

#!/bin/bash

CPU=$(top -bn1 | grep load | awk ‘{printf "%.2f\t\t\n", $(NF-2)}’)
MEMORY=$(free -m | awk ‘NR==2{printf "%.2f\t\t", $3*100/$2 }’)
DISK=`df -h | awk ‘$NF=="/"{ print $5}’ | sed ‘s/%//g’`

printf "{\"disk.utilization\" : %s , \"memory.utilization\" : %s , \"cpu.usage\" : %s}" "$DISK" "$MEMORY" "$CPU"

Output :

{"disk.utilization" : 23 , "memory.utilization" : 24.28 , "cpu.usage" : 0.64 }

Standard JSON output format

We recommend to adjust the script output to match one of the following JSON formats:

Include the scriptExceptions payload in the final JSON output is optional and should only be done if the script returns any exceptions. If no exceptions occur, the scriptExceptions payload should not be included in the final JSON output. The scriptExceptions payload can be used with any of the following formats. Below, we provide examples for a few of these formats. For more details on how to use scriptExceptions in RSE scripts, refer to the documentation.

Format 1:

Description:

This format is used when the metric does not have any dynamic components or instances. In such cases, construct the JSON by using the metric name as the key and mapping that key to the appropriate value.

Format for the JSON output when there are no exceptions:

{
  "Metric1": 98,
  "Metric2": 70,
  "Metric3": 80
}

Or

Format for the JSON output when exceptions occur:

{

  "Metric1": 98,
  "Metric2": 70,
  "Metric3": 80,
  "scriptExceptions": {
    "subject": "No monitoring data / Unable to fetch monitoring data / Incomplete script execution",
    "description": "Failed to collect data for following metrics. \n metricName: <Metric Name 1>, FailureReason: Failure Reason 1 \n metricName: <Metric Name 2>, FailureReason: Failure Reason 2",
    "raiseAlert": true,
    "logRequired": true,
    "alertState": "warning"
  }
}

Example:

{
  "system_windows_uptime_inMinutes": 120,
  "system_windows_overallCPU_utilization": 50,
}

Format 2:

Description: This format is used, when one metric returns a string and another metric returns a number. In such cases, a JSON output is prepared by mapping those strings or metrics to their appropriate values.

{
"Metric1": 98,
"Metric2": "STATE",
"Metric3": 80
}

Example:

{

   "system_windows_memory_usage": 35.75,
   "system_windows_winrm_status": "running"
}

Note: We recommend using numerical representation for String outputs by utilising Enumerated Map option of RSE.

Format 3:

Description: This format may be used, when the metric contains multiple components. In the below example, the data is fetched for multiple components of disks.

Format for the JSON output when there are no exceptions:

{
  "MetricName1": {
    "components": {
      "component1": 70,
      "component2": 98
    }
  },
  "MetricName2": {
    "components": {
      "component1": 77,
      "component2": 98
    }
  }
}

Or

Format for the JSON output when exceptions occur:

{
  "MetricName1": {
    "components": {
      "component1": 70,
      "component2": 98
    }
  },
  "MetricName2": {
    "components": {
      "component1": 77,
      "component2": 98
    }
  },

"scriptExceptions": {
    "subject": "No monitoring data / Unable to fetch monitoring data / Incomplete script execution",
    "description": "Failed to collect data for following metrics. \n metricName: <Metric Name 1>, FailureReason: Failure Reason 1 \n metricName: <Metric Name 2>, FailureReason: Failure Reason 2",
    "raiseAlert": true,
    "logRequired": true,
    "alertState": "warning"
  }
}

Example:

{
  "System_Windows_PhysicalDisk_WriteBytes_PerSec": {
    "components": {
      "physicaldisk_1": 0,
      "physicaldisk_2": 0
    }
  },
  "System_Windows_PhysicalDisk_AvgDisk_SecPerWrite": {
    "components": {
      "physicaldisk_1": 0,
      "physicaldisk_2": 0
    }
  }
}

Explanation:

In this example, System_Windows_PhysicalDisk_WriteBytes_PerSec is a metric name that has two components, physicaldisk_1 and physicaldisk_2, with their values mapped inside the components key.

Similarly, System_Windows_PhysicalDisk_AvgDisk_SecPerWrite is another metric name organized in the same way.

Format 4:

Description: This format is used when user needs metric-level alertTokens for all metrics of the script. Alert tokens are used to specify more information about the metrics in alert subject or in alert description or in both.

Format for the JSON output when there are no exceptions:

{
  "MetricName1": {
    "components": {
      "component1": "STATE",
      "component2": 98
    },
    "alertTokens": {
      "token1": "value",
      "token2": "value2"
    }
  },
  "MetricName2": {
    "components": {
      "component1": 77,
      "component2": 98
    },
    "alertTokens": {
      "token1": "value",
      "token2": "value2"
    }
  }
}

Or

Format for the JSON output when exceptions occur:

{
  "MetricName1": {
    "components": {
      "component1": "STATE",
      "component2": 98
    },
    "alertTokens": {
      "token1": "value",
      "token2": "value2"
    }
  },
  "MetricName2": {
    "components": {
      "component1": 77,
      "component2": 98
    },
    "alertTokens": {
      "token1": "value",
      "token2": "value2"
    }
  },
  "scriptExceptions": {
    "subject": "No monitoring data / Unable to fetch monitoring data / Incomplete script execution",
    "description": "Failed to collect data for following metrics. \n metricName: <Metric Name 1>, FailureReason: Failure Reason 1 \n metricName: <Metric Name 2>, FailureReason: Failure Reason 2",
    "raiseAlert": true,
    "logRequired": true,
    "alertState": "warning"
  }
}

Example:

{
  "system_linux_memory_utilization": {
    "components": {
      "real_memory_utilization": 50
    },
    "alertTokens": {
      "memory.usage": "Total memory: 16 GB, Used memory: 8 GB"
    }
  }
}

Explanation:

In this example, the system_linux_memory_utilization metric includes a component real_memory_utilization with a value of 50. The alert tokens offer a summary at the metric level, indicating “Total memory: 16 GB, Used memory: 8 GB”. This information is included in the alert description, the alert subject, or both for the system_linux_memory_utilization metric.

Format 5:

Description: This format is used, when user needs alert tokens at metric-level, for only few of the metrics of the script.

Format:

{
    "MetricName1": 254,
    "MetricName2": {
        "components": {
            "comp1": 90,
            "comp2": 60
        }
    },
    "MetricName3": {
        "components": {
            "component1": "STATE",
            "component2": 98
        },
        "alertTokens": {
            "token1": "value",
            "token2": "value2"
        }
    }
}

Example:

{
  "system_linux_services_status": {
    "components": {
      "iptables": "inactive",
      "irqbalance": "active",
      "kdump": "active",
      "ypxfrd": "inactive"
    },
    "alertTokens": {
      "services.active.counts": " 2 services are active",

      "services.inactive.counts": "2 services are inactive"
    }
  },
  "system_linux_totalServices_count": "4"
}

Explanation:

The system_linux_services_status metric includes two alert tokens at the metric-level.

• Token 1, denoted as services.active.counts, provides the count of active services.
• Token 2, identified as services.inactive.counts, offers the count of inactive services.

These tokens provide valuable insights into the status of the services monitored under this metric.

On the other hand, the metric system_linux_services_count does not have any associated alert tokens. So, alerts for this metric will not include any additional descriptive information.

Format 6:

Description: This format is used, when user needs Component Level Alert Tokens. Alert token value for each metric component are specified separately, as mentioned below:

Format for the JSON output when there are no exceptions:

{
  "MetricName1": 254,
  "MetricName2": {
    "components": {
      "comp1": 90,
      "comp2": 60
    }
  },
  "MetricName3": {
    "components": {
      "component1": "STATE",
      "component2": 98
    },
    "alertTokens": {
      "token1": "value",
      "token2": "value2"
    }
  },
  "MetricName4": {
    "components": {
      "component1": 10,
      "component2": 20
    },
    "alertTokens": {
      "token1": {
        "component1": "token 1 value of component 1",
        "component2": "token 1 value of component 2"
      },
      "token2": {
        "component1": "token 2 value of component 1",
        "component2": "token 2 value of component 2"
      }
    }
  }
}

Or

Format for the JSON output when exceptions occur:

{
  "MetricName1": 254,
  "MetricName2": {
    "components": {
      "comp1": 90,
      "comp2": 60
    }
  },
  "MetricName3": {
    "components": {
      "component1": "STATE",
      "component2": 98
    },
    "alertTokens": {
      "token1": "value",
      "token2": "value2"
    }
  },
  "MetricName4": {
    "components": {
      "component1": 10,
      "component2": 20
    },
    "alertTokens": {
      "token1": {
        "component1": "token 1 value of component 1",
        "component2": "token 1 value of component 2"
      },
      "token2": {
        "component1": "token 2 value of component 1",
        "component2": "token 2 value of component 2"
      }
    }
  },
  "scriptExceptions": {
    "subject": "No monitoring data / Unable to fetch monitoring data / Incomplete script execution",
    "description": "Failed to collect data for following metrics. \n metricName: <Metric Name 1>, FailureReason: Failure Reason 1 \n metricName: <Metric Name 2>, FailureReason: Failure Reason 2",
    "raiseAlert": true,
    "logRequired": true,
    "alertState": "warning"
  }
}

Example:

{
  "system_linux_interfaces_count": {
    "components": {
      "system_linux_interfaces_count": 8
    },
    "alertTokens": {
      "interfaces_names": "interfaces names are cni0, ens160, flannel.1, veth05f2cc15,  veth10f0079d,  veth42a5dd4f,  vetha7efbaa5,  vethdc3c2d78"
    }
  },
  "system_linux_network_interface_trafficIn": {
    "components": {
      "cni0": 14329842632,
      "ens160": 13491465976
    },
    "alertTokens": {
      "mac.address": "cni0:42-a5-a4-fd-86-eb,ens160:00-0c-29-b7-be-c3"
    }
  },
  "system_linux_network_interface_trafficOut": {
    "components": {
      "cni0": "9148429744",
      "ens160": "12488937472"
    }
  },

"scriptExceptions": {
    "subject": "An exception has occurred. Unable to fetch the monitoring data",
    "alertState": "critical",
    "description": "Failed to collect data for following metrics. \n metricName: system_linux_network_interface_errorsIn, FailureReason: rx_errors attribute not available for interface cni0, metricName: system_linux_network_interface_errorsOut, FailureReason: tx_errors attribute not available for interface cni0",
    "raiseAlert": true,
    "logRequired": true
  }
}

Explanation:

In this example, the JSON structure demonstrates the use of scriptExceptions along with component-level and normal metric-level alert token.

The system_linux_interfaces_count metric includes a component that counts the number of interfaces and a metric-level alert token listing the names of these interfaces.

For system_linux_network_interface_trafficIn, the components show the inbound traffic for specific interfaces, while the component-level alert token provides the MAC addresses of these interfaces, offering more details on the interface.

system_linux_network_interface_trafficOut presents outbound traffic data for the interfaces but does not include additional alert tokens.

Multiple Custom Script Arguments in Macro

Refer the document Multiple Custom Script Arguments for more details.

Generate Alert Tokens in RSE

Refer the document Generate Alert Tokens in RSE for more details.

Exception Handling In RSE

Refer the document Exception Handling In RSE for more details.

Enumerated Mapping In RSE

Refer the document Exception Handling In RSE for more details.

Next Steps

• See Add Filter and Define Threshold
• See Setting up Agentless Monitors
• See Agent-based vs Agentless monitoring