You must be running in FIPS mode. The settings will work fine in non-FIPS, but in FIPS mode if you even add the 1 setting of 'sslRootCAPath' your kvstore will fail. The only way to get kvstore to run in FIPS mode with your own certs is to rename your certs to be the default filepaths, which is $SPLUNK_HOME/etc/auth/server.pem and cacert.pem You will also have to cat 'appsCA.pem' to the end of your cacert.pem so that the Manage Apps UI works. Note that the kvstore is only needed on Search Heads, so you could disable or ignore it on non-SHs. The other problem you will still have is you cannot set 'requireClientCert=true' in FIPS mode, possibly in non-FIPS I have to confirm that again.   ... View more

On the IDX's server.conf you need to add this line in the [sslConfig] stanza: serverCert = /opt/splunk/etc/auth/mycerts/myCombinedServerCertificate.pem Then delete the sslPassword line from your server.conf if it's the default, Splunk will recreate it anyhow. That should fix it unless your cert is not prepared properly with just leaf cert + private key in the 'myCombinedServerCertificate.pem' __PRESENT __PRESENT ... View more

I believe this is another case of unclear documentation. The useSSL setting, as seen in the doc snippet you posted, does not say you don't need a cert, it says you don't need to set clientCert on the forwarder if the receiver has requireClientCert = false. In other words, the 'useSSL' setting on the forwarder is telling that forwarder to use TLS authentication, which is different than just encrypting your logs with TLS, which uses the TLS cert from the receiver. If you wish to encrypt your logs but don't need the receiver to require client TLS certs to authenticate, you don't need the useSSL=true setting. The other settings you listed such as check CN and SAN that the receiver cert matches the indexer you listed, are not required since you told the client to not require a server cert when connecting. So there are 3 related but distinct TLS topics here: log encryption using TLS, the forwarder authenticating the server using TLS, and the receiver authenticating the forwarder using TLS. The .conf.spec docs are not clear about which settings are for which TLS function, making it confusing. useSSL = <true|false|legacy> * Whether or not the forwarder uses SSL to connect to the receiver, or relies on the 'clientCert' setting to be active for SSL connections. * You do not need to set 'clientCert' if 'requireClientCert' is set to "false" on the receiver.   ... View more

If the intention is cloning all data to both and you're okay with the double license ingest, you just need to configure outputs similar to this below. There could be other TLS settings to include, but adding a comma-delimited list in [tcpout] will duplicate all logs to both groups listed, which can each have their own independent cert settings. Another method is to create a "050_clone_app" with just the [tcpout] stanza, calling the exact name of the tcp group in the 100 UF cloud app and your other outputs app to your on-prem. That way it's modular, can be managed with a DS, and when you're ready to cut one out you just delete the "050" app and the outputs you no longer want. We do this all the time to migrate from one Splunk to another with a clone period during migration and testing. outputs.conf [tcpout] defaultGroup = cloud_indexers, onprem_indexers [tcpout:cloud_indexers] server = 192.168.7.112:9998, idx2, idx3, etc clientCert = $SPLUNK_HOME/etc/auth/kramerCerts/SplunkServerCert.pem (retain settings from UF Cloud 100 app) [tcpout:onprem_indexers] server = 192.168.1.102:9998, idx2, idx3, etc clientCert = $SPLUNK_HOME/etc/auth/kramerCerts/SplunkServerCert.pem     ... View more

Your config looks good, so it could be the certs were not prepared correctly, or Splunk cannot read them. Splunk's docs are not clear or accurate for cert prep. The server cert on the Indexer must have the leaf cert followed by the private key and that's it. Any intermediate or root certs are simply referenced by the sslRootCAPath in server.conf Ensure those cert files are all readable and owned by the 'splunk' user and chmod them to 640 to be safe. Make sure you can cat the cert and root using the the splunk user on the indexer. By the way for log encryption, Splunk only uses the server cert (Indexer cert) to encrypt the logs. As others mentioned, use the openssl command and check the cert results from it. $SPLUNK_HOME/bin/splunk cmd openssl s_client -connect <your_indexer>:<port> -showcerts  Also search the internal logs on both the indexer and the UF for TLS errors. cat /opt/splunk/var/log/splunk/splunkd.log | grep -i 'tls\|ssl' ... View more

I don't think I can as a partner, but I frequently submit changes to their docs and post in Slack ... View more

You can just create a .../local/inputs.conf with stanzas and attributes that override the default config like this:   [monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log] _TCP_ROUTING = default-autolb-group index = _internal [monitor://$SPLUNK_HOME/var/log/splunk/metrics.log] _TCP_ROUTING = default-autolb-group index = _internal   ... View more

To add some clarity as that accepted answer is still quite vague or confusing... The easiest way would be to relate these field names to _time and _indextime recentTime = _indextime = last actual time this host was heard from by index(es) defined in metadata command, or in other more specific terminology, the last time it wrote logs to an index lastTime = _time = time stamp of the events from that host by index(es) defined in metadata command, in other words the latest timestamp in the set of events defined by the search ... View more

It is 12 years later, and this is still an issue. You cannot set 'requireClientCert=true' in server.conf on, for example, a Deployment Server, and have a working Web UI on that Deployment Server.  Setting 'requireClientCert=true' in server.conf still breaks the Web UI in late November 2024. ... View more

I would add that it's likely license usage would be greater for syslog ingested as HEC (being json) vs ingested as old school text log files. In that sense, SC4S would likely cause greater license usage than syslog, though you would save local disk capacity from having to store files until ingested. Just compare a text log file to it's json equivalent. ... View more

The csv lookup at index time is only to ingest  a csv and would not help in this case.  The iplocation command in SPL is actually a scripted lookup, but just like your scripted lookup suggestion, would still be at search time and would not help in this case.  ... View more

Each data source is different, but I noticed you tagged this for Windows so I'll post this guide: https://docs.splunk.com/Documentation/SplunkCloud/8.2.2112/Admin/WindowsGDI#Overview Essentially you login to the Splunk Cloud Search Head, download the Universal Forwarder app and you distribute that app to the /opt/splunkforwarder/etc/apps/ directory of the machines you want to send data to the Cloud. Depending on your needs and network architecture, it could get more complicated, but that is the simple version. So each Windows Server would need a Splunk UF (Universal Forwarder) and the Spunk Cloud UF app/ta/add-on (TA stands for Technical Add-on) to be able to send and collect data. Each data source also needs a configuration telling it what data to collect. This is often achieved by using a Splunk TA aka add-on on Splunkbase: https://splunkbase.splunk.com/ You can download the Splunk UF here: https://www.splunk.com/en_us/download/universal-forwarder.html For larger environments, the UF and required addons are usually distributed via a Splunk Deployment Server. Also, often data is sent through one or more Forwarders before Cloud to minimize firewall rules, or depending on your network architecture needs. All data sources need to be able to send data via tcp/9997 to Splunk Cloud. So the breakdown of steps is: Create an index on Splunk Cloud to receive your data Download the Cloud TA (called Cloud Universal Forwarder) from Splunk Cloud Search Head Install a UF and the Cloud TA onto your data source The Cloud TA needs to be untar'd to /opt/splunkforwarder/etc/apps/ Or it can be distributed via Splunk Deployment Server Install one or more add-ons aka TAs to /opt/splunkforwarder/etc/apps/ Configure and enable one or more 'inputs' or data to send by editing the inputs.conf within each TA/add-on There is usually a template inputs.conf in the default folder of each add-on. Create a /local folder (same level as /default) in each TA and copy that inputs.conf in there Edit it and enable one or more inputs to send data to Splunk There actually is an 'outputs.conf' but the Splunk Cloud TA/UF handles that to securely send to Splunk Cloud.   ... View more

Splunk only offers geoip via the command iplocation at search time. You could add the data using a 3rd part product before Splunk ingests it, but as far as Splunk doing that, the closest options you have (that I'm aware of) are: Custom Accelerated Data Model Modify Existing Accelerated Data Models to store those fields Create a lookup table - probably using KVSTORE due to high number of records It depends on your intended use cases for it. For the use cases you mentioned, it sounds like it would be used more for investigations in the original logs, in which case you'd have to tie it to time. For that purpose I would recommend using an Accelerated Data Model, though it wouldn't contain the raw logs. If you really need it in the raw logs, I would have either Logstash or CRIBL enrich the raw logs before ingest to Splunk or off to raw log storage depending on your needs.   ... View more

You will need a named capture group. If there is a space after the colon, or you're not sure, use this: message:\s*(?<raw_message>\S+) If there is always a space after the colon,  you could just use this below. The asterisk allow zero or more spaces (/s). You can learn more at regex101(dot)com or other sources. message: (?<raw_message>\S+) It's essential the same as the other person posted, but they were missing the ? for the named capture group and you don't really need anything before 'message' in this case.  ... View more

If using the FREE version at home, you will have to add the following line to your /etc/system/local/server.conf under the [general] tab (leave the other settings under [general] as well, but it should look like this, along with your other settings under there of course. For Splunk Enterprise, there is more information at the Docs link below. [general] allowRemoteLogin = always You can read about this and more information at this link, though I am submitting an edit to the page to correct typos and be more specific: https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/LaunchSplunkWeb ... View more

Use the second one I just listed for all destinations. If you want the status for each combination of src port and destination, you already had that in your image link. Note that your destinations were different which is why there were 2 states. The destinations are so similar perhaps you thought they were the same? ... View more

| stats latest(destination) as destination, latest(status) as status, by src port OR | stats values(destination) as destination, latest(status) as status by src port ... View more

Okay so you don’t want to split by destination then. Move destination to before the ‘by’ clause and use either latest(destination) or values(destination) if you want the whole list for that src and port. If you put destination after the BY clause, Splunk shows every unique combination of the fields after BY. ... View more

Dest is MPGDR vs MPGPR ... View more

I don’t see an issue in your image. The destinations are unique so that search shows the result of latest state per src dest combination. If that’s not what you wanted please list what you’d like to display instead. ... View more

I respectfully disagree. Though first(stats) is faster, it doesn’t guarantee the latest event chronologically, which is what the goal is here. First is faster because it doesn’t wait for the entire set of results, it will return the first event that streams in. Although that is often the most recent, it isn’t necessarily the most recent. ... View more

The only way that is possible is something else is wrong with either the data or your environment. Please post your entire SPL and a screenshot of results. As I mentioned, either the values aren’t actually unique or there are many events at once and an issue with duplicate events on your indexers. ... View more

The way to get the most recent status would be to end your query with: | stats latest(status) by src destination port If you wish to also see the time of the event in the results you would add it as a value rather than split the stats command by _time, to avoid seeing every entry for that src/dest/port combo, by adding this to the end of your query instead: | stats values(_time) as _time, latest(status) by src destination port If either of these result in more than one status, there is something else wrong with the values that are in src, destination, and/or port. The only ways you would see more than 1 status for these is if: 'status' field has multiple values per event (perhaps due to commands earlier in your search The values are unique between src, destination, port You can test if the 'status' field has multiple values using this command: | eval n = mvcount(status) | where n>1 | sort - n I would also note that 'destination' is not a CIM-compliant field, it should be normalized to just 'dest' using an alias or other method. Hope this helps! ... View more