The Client forwarder management not showing the clients

[AAlhabba]

Dears,

       After upgraded Splunk from 9.1.2 version to 9.2.0 version, the deployment server not showing the clients, but Splunk receiving logs from clients, and also the client agents showing on all Splunk servers under setting --> Forwarder Managment except Deployment server, I don't know how that occurred, I didn't change anything.

Kindly your support for that.

Best Regards, 

[askargbo]

I have tried all the recommendations in this thread and non of them works. I upgraded from 9.0 to 9.3, but the clients are not phoning in.

[JoshuaJJ]

This is not working for me 😞 

I have the app in place on my DS (/etc/apps/DS_Fix/local/outputs.conf 

[kell_cena]

@AAlhabba , thank you for the solution .Worked like a charm.

[AJ_splunk1]

Does this work with splunk cloud as well? WE have splunk onprem deployment server, indexers are all in the cloud and experiencing the same where clients are not showing up after an update to 9.2.x. They are phoning home however as per the logs

[AJ_splunk1]

nvm figured it out. It was the output.conf in this app - etc/apps/SplunkDeploymentServerConfig. Documentation is a bit confusing for this

[learningmode]

Hi @AJ_splunk1, just a heads-up that https://docs.splunk.com/Documentation/Splunk/9.2.0/Updating/Upgradepre-9.2deploymentservers states that " In particular, there is a new system-generated app, etc/apps/SplunkDeploymentServerConfig, which contains configuration files necessary to the proper functioning of the deployment server. Do not alter this directory or its files in any way."

I chose to implement the 9.2 fix (also shown on the page link above) as a separate app in \etc\apps on the ds and just called something like "Fix_DSClientList" so it's more obvious there's a modification in place.

I hope that helps.

[michael78]

In my case I had this configuration line in another app:

forwardedindex.2.whitelist = (_audit|_introspection|_internal)

I´ve recognized that in etc/apps/SplunkDeploymentServerConfig/default/outputs.conf there is statement

forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker|_dsclient|_dsphonehome|_dsappevent)

My took preference because of being in local. After I have removed my local one I am able to see clients in GUI of deployment server again.

[isoutamo]

The best ways to solve this kind of issues is use btool like

splunk btool outputs list --debug 

This shows all stanzas and values as splunk takes those into use after rebooting splunkd if those are added after last reboot into files. 

[AAlhabba · Accepted solution]

Dears,

        I have resolved the issue by adding below configuration under outputs.conf in the deployment server, then restart splunk service in the deployment server.

[indexAndForward]

index = true

selectiveIndexing = true

You can see below URL:

Upgrade pre-9.2 deployment servers - Splunk Documentation

[andrewtrobec]

Boy am I glad to have found this thread.  Got my problem solved, thank you so much ❤️

[JRW]

Another fix to try is as follows:

Find your distsearch.conf and then find the stanza that has default = true in it. In that stanza, make sure localhost:localhost is listed in the setting below

servers = 

For example, it was like this before:

distributedSearch:testgroup1
default = true
servers = somehostname.company.com

Once you find that stanza, add localhost to make it look like this (and it's literal in that it's simply localhost:localhost)

distributedSearch:testgroup1
default = true
servers = somehostname.company.com, localhost:localhost

Restart the DS and from the internal thread within a few minutes/hours the clients should start to populate again

[yomesky2000]

@JRW You solution worked for me like charm. I spend more than 6 hours troubleshooting until i stumble on yours and decided to try it out even though its not marked as the preferred solution.

Thank you

[Laeghaire]

THIS is what worked for me

I did add the selective indexing stanz before but that alone was not enough

thx so much, I would have never EVER guessed to add that there 

[JoshuaJJ]

This worked instantly!  I appreciate you! 

Thanks, 

JJJ 

[jmcnee]

This was so helpful and fixed my problem, thankyou very much!

[PickleRick]

Thereare no miracles. If they are showing in the Forwarder Management section on a server different than your designated DS, they must have been pointed there somehow. Check your deployment server definition on your forwarders.

[landrujw]

We are experiencing the same thing. The clients are showing up in the client_events logs checking in and phoning home on the deployment server. But after updating to 9.2 they aren't appearing under the Settings>Forwarder Management page on the DS. We have not made any changes to the forwarders yet.

[mykol_j]

FWIW, happening here as well, with 9.2.0.1.

Checked all The Things mentioned in that doc everyone keeps referencing, including those stanzas mentioned numerous times here.

Another symptom of mine is that the ForwarderManager (deployer) doesn't appear in my monitored servers in the SplunkManager (aka Master).

[mykol_j]

This is nuts. Go figure.

I ended up fixing this my removing the "Deployment Server" role from the system, saving it, then adding it back, restarting the service, bam! Fixed.

I'd rather be lucky than good...

[JoshuaJJ]

Hmmm, are you talking about the role defined within the Monitoring Console? I am having tons of issues resolving this.