Hello everyone! I came across a strange behavior. I was building a dashboard and noticed that some results look unexpected. The results are presented at the top of the screenshot. On the last row, you can see that ProvDuration is 0. Also, StartTime and EndTime are equal. Moreover, other fields are also equal, and it's illogical due to the search specifics. As you can see, StartTime and EndTime represent the min and max values of the _time field.   index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP") | rex field=_raw "VM\s+(?<MachineName>.*)$" | table _time, PoolId, MachineName, _raw | transaction MachineName startswith="Pool" endswith="startup" maxevents=2 keeporphans=false | search (PoolId="*") (MachineName="*") | search duration<=700 | stats min(duration) AS DurationMin, avg(duration) AS DurationAvg, max(duration) AS DurationMax, min(_time) AS StartTime, max(_time) AS EndTime BY PoolId | eval DurationMin = round(DurationMin, 2) | eval DurationAvg = round(DurationAvg, 2) | eval DurationMax = round(DurationMax, 2) | eval ProvDuration = round((EndTime - StartTime), 2) | eval StartTime = strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3Q") | eval EndTime = strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3Q") | table PoolId, DurationMin, DurationAvg, DurationMax, ProvDuration, StartTime EndTime   I decided to dig deeper and try to analyze the search more carefully. After I moved to the search through the dashboard, I found that the search results look different. The last row looks as it should be. You can see these results at the bottom of the screenshot. What could be wrong with my search, and what am I missing? ... View more

Hello to everyone! I'm not sure how to correctly name this thing, but I will carefully try to explain what I want to achieve. In our infrastructure we have plenty of Windows Server instances with Universal Forwarder installed. All servers are divided into groups according to the particular application that the servers host. For example, Splunk servers have group 'spl,' remote desktop session servers have group 'rdsh,' etc. Each server has an environment variable with this group value. By design, the access policy to logs was built on these groups. One group - one index. Because of it, each UF input stanza has the option "index = group.". According to this idea, introspection logs of UF agents are related to the SPL (or Splunk) group\index. And here the nuisance started. Sometimes UF agents report about errors that demand some things on the running hosts, for example, restarting the agent manually. I see these errors because I have access to the 'spl' index, but I don't have access to all Windows machines and I have to notify the machine owner about it manually. So, the question is how to create a sort of tag or field on the UF that can help me separate all Splunk UF logs by these groups? Maybe I can use our environment variable to achieve it? I only need to access this field during search time to create various alerts that will notify machine owners instead of me. ... View more

Hello to everyone! I want to build a dashboard with which I can access information from config files of indexer cluster I know that the typical scenario to access config files is using REST endpoints "/services/configs/conf-*" But as I understood, these endpoints show only configuration files stored under /system/local/*.conf Is it a way to access config files stored under /manager-apps/local ? ... View more