This may be a "dumb" question, but I'll just throw it out there while I try to work it out. The Python for Scientific Computing (PSC) app is HUGE. We have a clustered environment and Search Heads (SH) receive configuration from our deployer. During initial setup the maximum bundle size was increased to allow pushing the PSC app from the deployer to the SHs. While it worked we've noticed that any push after adding the PSC app to the deployer now takes around 2 minutes to complete, regardless of how small of a change even if no restart is needed. I was hoping there was a way to install the PSC app locally in the search head cluster without going through the deployer. To option to "install from file" is not present in the web UI, assumably since we have a deployer for managing apps. Removing the app from the deployer and consequentially the SH cluster I tried to unpack the app into the /opt/splunk/etc/apps folder but it is then removed/deleted automatically as the cluster is restarted. Presumably since it is not available on the deployment server. So, how should we install and use PSC in a clustered environment? Is the only/correct way to to push the giant app from the deployer or is there another way to distribute the app?  All feedback and/or suggestions are welcome ... View more

Without a tiered storage model it seems like there would be little argument for using cold/frozen storage. Except potentially if additional compression helps save space. If not, using only a homePath in indexes.conf would seem like it would make all data readily accesible as hot/warm. However, checking the documentation there seems to be three paths for indexes that are reqired for Splunkd to start being homePath, coldPatch and thawedPath. (indexes.conf - Splunk Documentation) So using a single disk/volume/mount, what does the inputs.conf look like? Should the same path just be set for all three? Making sure that maxVolumeDataSizeMB adds up to the total volume available on /data/splunk/warm. [volume:storage] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 2800000 ... ... [volume:_splunk_summaries] path = /data/splunk/warm/ # ~ 200GB maxVolumeDataSizeMB = 200000 ... ... [main] homePath = volume:storage/defaultdb/db coldPath = volume:storage/defaultdb/colddb thawedPath = $SPLUNK_DB/defaultdb/thaweddb [history] homePath = volume:storage/historydb/db coldPath = volume:storage/historydb/colddb thawedPath = $SPLUNK_DB/historydb/thaweddb [summary] homePath = volume:storage/summarydb/db coldPath = volume:storage/summarydb/colddb thawedPath = $SPLUNK_DB/summarydb/thaweddb ... ... [windows] homePath = volume:storage/windows/db coldPath = volume:storage/windows/colddb summaryHomePath = volume:storage/windows/summary thawedPath = $SPLUNK_DB/windows/thaweddb tstatsHomePath = volume:_splunk_summaries/windows/datamodel_summary frozenTimePeriodInSecs = 63072000 [linux] homePath = volume:storage/linux/db coldPath = volume:storage/linux/colddb summaryHomePath = volume:storage/linux/summary thawedPath = $SPLUNK_DB/linux/thaweddb tstatsHomePath = volume:_splunk_summaries/linux/datamodel_summary frozenTimePeriodInSecs = 63072000   I'm assuming this would work, right? Though as it seems that Splunk requires, and does make use of "cold" and "thawed" anyway, does it make more sence to just partition mounts for warm and cold separately anyway?  [volume:warm] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 500000 [volume:cold] path = /data/splunk/warm/ # adjust when correct disk is mounted maxVolumeDataSizeMB = 2500000 ... ... [volume:_splunk_summaries] path = /data/splunk/warm/ # ~ 200GB maxVolumeDataSizeMB = 200000 ... ... [main] homePath = volume:warm/defaultdb/db coldPath = volume:cold/defaultdb/colddb thawedPath = $SPLUNK_DB/defaultdb/thaweddb [history] homePath = volume:warm/historydb/db coldPath = volume:cold/historydb/colddb thawedPath = $SPLUNK_DB/historydb/thaweddb [summary] homePath = volume:warm/summarydb/db coldPath = volume:cold/summarydb/colddb thawedPath = $SPLUNK_DB/summarydb/thaweddb ... ... [windows] homePath = volume:warm/windows/db coldPath = volume:cold/windows/colddb summaryHomePath = volume:warm/windows/summary thawedPath = $SPLUNK_DB/windows/thaweddb tstatsHomePath = volume:_splunk_summaries/windows/datamodel_summary frozenTimePeriodInSecs = 63072000 [linux] homePath = volume:warm/linux/db coldPath = volume:warm/linux/colddb summaryHomePath = volume:warm/linux/summary thawedPath = $SPLUNK_DB/linux/thaweddb tstatsHomePath = volume:_splunk_summaries/linux/datamodel_summary frozenTimePeriodInSecs = 63072000 Does it matter and what would be "best praxis".  ... View more

This is not a particulary crucial question but it has been nagging me for a while. When applying changes to indexes.conf on the manager node I usually do a calidate -> show -> apply round. But I am somewhout confused about a detail. When you validate a bundle after making modifications you get an updated "last_validated_bundle" checksum. In my mind, when you then apply this bundle the now "last_validated_bundle" checksum should become the new "latest_bundle" and "active_bundle". But this is not the case.  $ splunk apply cluster-bundle Creating new bundle with checksum=<does_not_match_last_validated_bundle> Applying new bundle. The peers may restart depending on the configurations in applied bundle. Please run 'splunk show cluster-bundle-status' for checking the status of the applied bundle. OK After the new bundle has been applied active_bundle, latest_bundle and last_validated_bundle checksums all match again. But why is the checksum produced after bundle validation not matching the checksum for the applied bundle? Have a great weekend! ... View more

I get weekly email updates with results from weekly URA scans. After noticing that we had outdated apps we rolled out updates for three public apps, Sankey Diagram, Scalable Vector Graphics and Splunk Dashboard Examples. In our testing environment URA is now content and all apps pass jQuery scans without issues. However, in our production environment URA scan still fails in all three apps. It does not specify which files or of there is a problem om one or all instances so I don’t know what is causing the results. I have double and triple checked the apps comparing hash values for every file both on the deployment server and on all individual test and production search heads. Everything except for the “install hash” in “meta.local” is identical in both test and production environment. Apps are all identical between cluster members in test and production environment respectively. There are not additional files present on any search head in the production environment. Why is URA still failing these apps only in the production environment? How can I identify the reason for the scan failures as I they should all pass in both environments, being identical and all. Any and all suggestions are most welcome All the best ... View more

I'm using the Splunk TA for linux to collect serverlogs. Some background Looking in the "_internal" log I am seing a lot of these errors: 08-23-2024 15:52:39.910 +0200 WARN DateParserVerbose [6460 merging_0] - A possible timestamp match (Wed Aug 19 15:39:00 2015) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source=lastlog|host=<hostname>|lastlog|13275 08-23-2024 15:52:39.646 +0200 WARN DateParserVerbose [6460 merging_0] - A possible timestamp match (Fri Aug 7 09:08:00 2009) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source=lastlog|host=<hostname>|lastlog|13418 08-23-2024 15:52:32.378 +0200 WARN DateParserVerbose [6506 merging_1] - A possible timestamp match (Fri Aug 7 09:09:00 2009) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source=lastlog|host=<hostname>|lastlog|13338 This is slightly confusing and somewhat problematic as  the "lastlog" is collected not through a filewatch but from scripted output. The "lastlog" file is not collected/read and a stats-check on the file confirms accurate dates. However, this is not the source of the problem. I cannot se anything in the output from the commands in the script (Splunk_TA_nix/bin/lastlog.sh) which would indicate the precense of a "year"/timestamp. The indexed log does not contain "year" and the actual _time timestamp is correct. These "years" in "_internal" are also from a time when the server was not running/present, so they are not collected from any actual source "on the server". And the questions - Why am I seeing these errors - From where are these problematic "timestamps" generated - How do I fix the issue All the best   ... View more