Universal Forwarders not forwarding windows performance monitor inputs

[sgarvin55]

We are experiencing issues where some of our systems which appear to be configured properly will not forward certain windows performance monitor counters. The forwarders are controlled by a deployment server, and appear to be contacting the deployment server correctly, and forwarding windows event log events properly, but some or all performance monitor inputs do not get forwarded.

Example of perfmon input in inputs.conf:

[perfmon://Network Interface]
_rcvbuf = 1572864
counters = Bytes Received/sec;Bytes Sent/sec
disabled = 0
evt_dc_name =
evt_dns_name =
evt_resolve_ad_obj = 0
host = $decideOnStartup
index = perfmon
instances = *
interval = 20
mode = multikv
object = Network Interface

One of the error messages we see in the splunkd.log after enabling DEBUG is:

DEBUG ExecProcessor - Ran script: "D:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe", took 6.345292 seconds to run, 0 bytes read, exited with code -1

[sgarvin55 · Accepted solution]

Once the customer got in touch with Microsoft, he was able to narrow it down to the "Dfsr" performance counter using exctrlst.exe (see: http://blogs.technet.com/b/askperf/archive/2010/03/05/two-minute-drill-disabled-performance-counters... ). Once he disabled that counter all of their normal counters returned, and are now being forwarded correctly.