KVStore does not start after upgrading to Splunk Enterprise 9.4.0

[Lekapu]

Well i spend hours trying to get this working, i tried the following first with no luck:

- CPU AVX support    [is supported]  
- Removed mongo.lock
- renamed server.pem

The following resolved my problems:

changed "sslVerifyServerName" from "True" to "False" under server.conf , restarted and kvstore came up.

Current Versions on  my Search Head:

MongoDB = db version v7.0.14

Splunk Version = Splunk 9.4.0

KvStore serverVersion = 4.2.25

For some reason, KVStore did not upgrade during Splunk upgrade. I will look at upgrading KVStore to version 7 later on.

[Lekapu]

I come back after sweating for hours on this, I could not go ahead with upgrading the entire Splunk Estate to version 9.4 before upgrading kvstore to version 7.  First of all, thanks to @etoombs  and @livehybrid for the guidance.

Below is what helped me:

1. I believe kvstore upgraded the first time when i upgraded Splunk Enterprise to 9.4 on the Search Head, as i had "storageEngineMigration = true" under the [kvstore] stanza, but because of TLS issues, it could not load.

2. After upgrading to Splunk 9.4, i managed to bring kvstore up by setting "sslVerifyServerName = False" under the [kvstore] stanza, but it came back as version 4.2

3. To upgrade kvstore to version 7, i ran "./splunk start-standalone-upgrade kvstore -version 7.0 -dryRun true" and tailed logs from "/opt/splunk/var/log/splunk/splunkd.log" and all was well.

4. When i ran the actual kvstore upgrade command, i got the following error "KVStore Upgrade is not supported on this instance! Reason=Instance is configured with custom KVStore certificates". and once i applied "sslVerifyServerCert = False", and restarted splunk, kvstore came back as version 7

My current server.conf settings:

[sslConfig]
sslPassword = ****************************************
sslVersions = tls1.2
serverCert = /opt/splunk/etc/auth/server.pem
sslRootCAPath = /opt/splunk/etc/auth/cacert.pem
sslVerifyServerName = False
cliVerifyServerName = False

[kvstore]
serverCert = /opt/splunk/etc/auth/server.pem
sslRootCAPath = /opt/splunk/etc/auth/cacert.pem
sslVersions = tls1.2
sslPassword = ****************************************
sslVerifyServerCert = False
sslVerifyServerName = False
storageEngine = wiredTiger
#storageEngineMigration = true

The below configs did not help me with anything, i was trying to get rid of this error " WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details"

sslVerifyServerName = False
cliVerifyServerName = False

kvstore and MongoDB version

featureCompatibilityVersion : 7.0

serverVersion : 7.0.14

db version v7.0.14

N.B: We are running default self-signed certs. For UI access we have 3rd party CA signed certs loaded on a reverse proxy. As a result, i did not mess with web.conf settings.

[etoombs]

We had some issues with the upgrade also. Here are the checks we did:

• Verify that certificates are valid and you have the full certificate chain (or revert to default splunk certs for the duration of the upgrade)
• Run btool on server.conf and look for storageEngineMigration under the [kvstore] stanza. If this is present in a local file, comment it out.
• Try running a test of the kvstore upgrade :  /opt/splunk/bin/splunk start-standalone-upgrade kvstore -version 7.0 -dryRun true - this may return errors. 
• Run the actual upgrade (remove the dry run)
  • When you run the upgrade, you can monitor /var/log/splunk/mongod_upgrade.log - You'll be able to see it upgrading through multiple versions from the 4.2 up to 7. 

[livehybrid]

Hi @Lekapu 

What are the logs relating to mongo/kvstore in $SPLUNK_HOME/var/log/splunk/splunkd.log? 

This might help us look into this further.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will