DB connection errors- no Http Event Collectors available

[juhiacc]

Hi,
We have db connect connections & inputs created in Splunk HF. We see that it has status=FAILED sometimes and below is the error captured through internal  DB logs.

Logs: /opt/splunk/var/log/splunk/splunk_app_db_connect_job_metrics.log

 /opt/splunk/var/log/splunk/splunk_app_db_connect_server.log

Error-
ERROR org.easybatch.core.job.BatchJob - Unable to write records java.io.IOException: There are no Http Event Collectors available at this time.

Can someone help?

[PickleRick]

I don't recall ever seeing dbconnect configured so that it sends to a HEC input outside of the HF it's running on. Theoretically it's possible - see https://docs.splunk.com/Documentation/DBX/3.18.2/DeployDBX/settingsconfspec but I must say I've never seen it configured this way.

Anyway, first check your config., then debug apropriate HEC inputs.

[livehybrid]

Hi @juhiacc 

The error java.io.IOException: There are no Http Event Collectors available at this time indicates that the Splunk DB Connect application, running on your Heavy Forwarder (HF), cannot successfully send data to the configured HTTP Event Collector (HEC) endpoint(s) on your Splunk indexers.

This usually stems from one of the following issues:

1. HEC Not Enabled or Misconfigured on Indexers:

   • Verify that HEC is enabled globally and available on all indexers/HFs which DB Connect is directed to.
   • Confirm the specific HEC token used by your DB Connect input is enabled and valid on all appropriate hosts.
   • Ensure the index specified in the HEC token configuration and/or the DB Connect input exists and is not disabled.

2. Incorrect HEC Configuration in DB Connect:

   • Within the DB Connect App on the HF, Ensure HEC is configured correctly,

3. Network Connectivity Issues:

   • Confirm the HF can reach the indexer(s) on the HEC port (default 8088). Check firewalls between the HF and indexers.
   • Use tools like curl or telnet from the HF to test connectivity to https://<indexer_hostname_or_IP>:8088.
   • If using a load balancer in front of your indexers for HEC, ensure it is configured correctly and all backend indexer nodes are healthy and responding.

4. Indexer(s) Overloaded or Unavailable:

   • Check the health of your indexers using the Monitoring Console (Monitoring Console -> Indexing -> Performance -> Indexer Performance). Overloaded indexers might refuse HEC connections.
   • Ensure the indexers are running and accessible.

Additional tips:

• If you have multiple indexers or an indexer cluster, ensure the HEC configuration is consistent across all relevant nodes.
• If using deployment server to manage the DB Connect app configuration on the HF, ensure the correct config is deployed
• Check splunkd.log on both the HF and the target indexer(s) for more detailed connection errors or HEC processing issues around the time the DB Connect job fails.

Relevant Documentation worth checking:

• Splunk DB Connect Overview: https://docs.splunk.com/Documentation/DBX/latest/DeployDBX/AboutSplunkDBConnect
• Set up and use HTTP Event Collector in Splunk Web: https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector
• Troubleshooting DB Connect: https://docs.splunk.com/Documentation/DBX/3.18.2/DeployDBX/Troubleshooting 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[kiran_panchavat]

@juhiacc 

If you have trouble receiving data from DBX3, search the internal index for

xx-xx-xxxx xx:xx:xx.xxx +0000 FATAL HTTPServer - Could not bind to port 8088

you could do:

netstat -anp | grep 8088

And ensure that you see the Splunk process using the port number.

You might also see the port 8088 in your metrics.log file of your Splunk server receiving the traffic if there is data coming through...

• Is there a firewall between your DB connect server and the HEC server?
• Ensure the port(s) are availble
• Ensure on Splunk HEC server, you have global settings enabled:

Click Settings > Data Inputs.
Click HTTP Event Collector.
Click Global Settings.
In the All Tokens toggle button, select Enabled.

• Some other aspects to check and troubleshoot:

#Check if the Hec collector is healthy

curl -k -X GET -u admin:mypassword https://MY_Splunk_HEC_SERVER:8088/services/collector/health/1.0

#Check if HEC stanzas with config are configured

/opt/splunk/bin/splunk http-event-collector list -uri https://MY_Splunk_HEC_SERVER:8089

#Check the settings using btool

/opt/splunk/bin/splunk cmd btool inputs list --debug http