Hi All
I have recently taken over the admin of our splunk server, I upgraded to 7.2.0 and its been running fine for a while, yesterday we started getting errors:
Failed to start KV Store process. See mongod.log and splunkd.log for details.
11/13/2018, 9:09:18 AM
KV Store changed status to failed. KVStore process terminated.
11/13/2018, 9:09:16 AM
KV Store process terminated abnormally (exit code 62, status exited with code 62). See mongod.log and splunkd.log for details.
11/13/2018, 9:09:16 AM
after looking that up I saw that the internal SSL cert had expired so I renewed it as per the instructions:
"set OPENSSL_CONF=D:\Splunk\openssl.cnf
D:\Splunk\etc\auth>d:\splunk\bin\splunk createssl server-cert -d . -n server"
This is now showing the cert to be valid. But now, I am getting the error below in the mongod log file.
2018-11-13T09:09:16.227Z W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] MongoDB starting : pid=8104 port=8191 dbpath=E:\Splunk\var\lib\splunk\kvstore\mongo 64-bit host=PRDSPLKAPP02
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] db version v3.6.7
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] git version: 2628472127e9f1826e02c665c1d93880a204075e
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2o-fips 27 Mar 2018
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] allocator: tcmalloc
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] modules: none
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] build environment:
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] distmod: 2008plus-ssl
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] distarch: x86_64
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] target_arch: x86_64
2018-11-13T09:09:16.379Z I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, ssl: { PEMKeyFile: "E:\Splunk\etc\auth\server.pem", PEMKeyPassword: "", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." } }, replication: { oplogSizeMB: 200, replSet: "C3E895A2-5F0A-4968-856E-C1C0047199B9" }, security: { javascriptEnabled: false, keyFile: "E:\Splunk\var\lib\splunk\kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "E:\Splunk\var\lib\splunk\kvstore\mongo", engine: "mmapv1", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }
2018-11-13T09:09:16.401Z I JOURNAL [initandlisten] journal dir=E:\Splunk\var\lib\splunk\kvstore\mongo\journal
2018-11-13T09:09:16.401Z I JOURNAL [initandlisten] recover : no journal files present, no recovery needed
2018-11-13T09:09:16.457Z I JOURNAL [durability] Durability thread started
2018-11-13T09:09:16.458Z I JOURNAL [journal writer] Journal writer thread started
2018-11-13T09:09:16.460Z I CONTROL [initandlisten]
2018-11-13T09:09:16.460Z I CONTROL [initandlisten] ** WARNING: No SSL certificate validation can be performed since no CA file has been provided
2018-11-13T09:09:16.460Z I CONTROL [initandlisten] ** Please specify an sslCAFile parameter.
2018-11-13T09:09:16.488Z F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.
2018-11-13T09:09:16.488Z I NETWORK [initandlisten] shutdown: going to close listening sockets...
2018-11-13T09:09:16.488Z I REPL [initandlisten] shutdown: removing all drop-pending collections...
2018-11-13T09:09:16.488Z I REPL [initandlisten] shutdown: removing checkpointTimestamp collection...
2018-11-13T09:09:16.488Z I REPL [initandlisten] shutting down replication subsystems
2018-11-13T09:09:16.488Z W REPL [initandlisten] ReplicationCoordinatorImpl::shutdown() called before startup() finished. Shutting down without cleaning up the replication system
2018-11-13T09:09:16.488Z I STORAGE [initandlisten] shutdown: waiting for fs preallocator...
2018-11-13T09:09:16.488Z I STORAGE [initandlisten] shutdown: final commit...
2018-11-13T09:09:16.492Z I JOURNAL [initandlisten] journalCleanup...
2018-11-13T09:09:16.492Z I JOURNAL [initandlisten] removeJournalFiles
2018-11-13T09:09:16.497Z I JOURNAL [initandlisten] old journal file E:\Splunk\var\lib\splunk\kvstore\mongo\journal\j._0 will be reused as E:\Splunk\var\lib\splunk\kvstore\mongo\journal\prealloc.0
2018-11-13T09:09:16.498Z I JOURNAL [initandlisten] Terminating durability thread ...
2018-11-13T09:09:16.521Z I JOURNAL [journal writer] Journal writer thread stopped
2018-11-13T09:09:16.521Z I JOURNAL [durability] Durability thread stopped
2018-11-13T09:09:16.521Z I STORAGE [initandlisten] shutdown: closing all files...
2018-11-13T09:09:16.534Z I STORAGE [initandlisten] closeAllFiles() finished
2018-11-13T09:09:16.534Z I STORAGE [initandlisten] shutdown: removing fs lock...
2018-11-13T09:09:16.535Z I CONTROL [initandlisten] now exiting
2018-11-13T09:09:16.535Z I CONTROL [initandlisten] shutting down with code:62
The two big errors being:
"Please specify an sslCAFile parameter."
where do I specify this?
** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.
would this not have upgraded with the version of Splunk? if not, how do I upgrade this?
any help would be appreciated, Thank you
... View more
