Anomali ThreatStream App not processing snapshots from API?

[guarisma]

Hello,

We've setup our Splunk Search Head to download snapshots from ThreatStream API directly, while troubleshooting, we observed that it was downloading the snapshots from hxxps://ts-optic.s3.amazonaws.com/snapshots/... but then had issues processing it.

2022-11-03 02:01:47,394 18860 ERROR threatstream_app - threatstream_kvstore> Autologin succeeded, but there was an auth error on next request. Something is very wrong.
2022-11-03 02:01:47,443 18860 ERROR threatstream_app - threatstream_kvstore> Failed at add_kvs_batch - sz == 1, collection_name: ts_md5, data: [{'date_last': '2016-02-21T14:52:32.000Z', 'id': '0', '_key': '99929352'}]
2022-11-03 02:01:47,443 18860 ERROR threatstream_app - threatstream_kvstore> Autologin succeeded, but there was an auth error on next request. Something is very wrong.
2022-11-03 02:01:47,464 18860 ERROR threatstream_app - threatstream_kvstore> Failed at add_kvs_batch - sz == 1, collection_name: ts_md5, data: [{'date_last': '2016-02-21T14:52:37.000Z', 'id': '0', '_key': '99929603'}]
2022-11-03 02:01:47,464 18860 ERROR threatstream_app - threatstream_kvstore> Autologin succeeded, but there was an auth error on next request. Something is very wrong.
2022-11-03 02:01:48,677 18860 INFO threatstream_app - ioc_loader> 193571 items with id="0" saved to kvs: ts_md5 for deletion, time: 35505.908512592316
2022-11-03 02:01:48,678 18860 INFO threatstream_app - ioc_loader> 193571 items with id="0" saved to kvs: ts_md5 for deletion, time: 35505.908512592316
2022-11-03 02:01:49,059 18860 ERROR threatstream_app - ts_ioc_ingest> failed to download optic intelligence: Autologin succeeded, but there was an auth error on next request. Something is very wrong.
2022-11-03 02:01:49,059 18860 ERROR threatstream_app - ts_ioc_ingest> failed to download optic intelligence: Autologin succeeded, but there was an auth error on next request. Something is very wrong.
2022-11-03 02:01:49,933 18860 ERROR threatstream_app - ts_ioc_ingest> Traceback (most recent call last):
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 290, in wrapper
    return request_fun(self, *args, **kwargs)
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 71, in new_f
    val = f(*args, **kwargs)
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 622, in delete
    response = self.http.delete(path, self._auth_headers, **query)
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 1169, in delete
    return self.request(url, message)
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 1255, in request
    raise HTTPError(response)
splunklib.binding.HTTPError: HTTP 401 Unauthorized -- call not properly authenticated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 232, in _handle_auth_error
    yield
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 301, in wrapper
    return request_fun(self, *args, **kwargs)
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 71, in new_f
    val = f(*args, **kwargs)
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 622, in delete
    response = self.http.delete(path, self._auth_headers, **query)
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 1169, in delete
    return self.request(url, message)
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 1255, in request
    raise HTTPError(response)
splunklib.binding.HTTPError: HTTP 401 Unauthorized -- call not properly authenticated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/splunk/etc/apps/threatstream/bin/ts_ioc_ingest.py", line 284, in download_iocs
    TmDataManager(splunka=remote_splunk, logger=logger).process_data()
  File "/opt/splunk/etc/apps/threatstream/bin/ts/tm_data_manager.py", line 176, in process_data
    self._process_data()
  File "/opt/splunk/etc/apps/threatstream/bin/ts/tm_data_manager.py", line 245, in _process_data
    self.load_from_lookup_files()
  File "/opt/splunk/etc/apps/threatstream/bin/ts/tm_data_manager.py", line 508, in load_from_lookup_files
    iocs.load_iocs()
  File "/opt/splunk/etc/apps/threatstream/bin/ts/lookup_iocs.py", line 404, in load_iocs
    util.utils.remove_0_id_values(self.kvsm, kvs)
  File "/opt/splunk/etc/apps/threatstream/bin/util/utils.py", line 143, in remove_0_id_values
    remove_delete_id_values(kvsm, ioc_kvs_name, 'id', '0')
  File "/opt/splunk/etc/apps/threatstream/bin/util/utils.py", line 146, in remove_delete_id_values
    kvsm.delete_kvs(kvs, {id_name : delete_id_value})
  File "/opt/splunk/etc/apps/threatstream/bin/util/kvs_manager.py", line 286, in delete_kvs
    collection.data.delete(query=json.dumps(query_dict))
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/client.py", line 3678, in delete
    return self._delete('', **({'query': query}) if query else {})
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/client.py", line 3631, in _delete
    return self.service.delete(self.path + url, owner=self.owner, app=self.app, sharing=self.sharing, **kwargs)
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 301, in wrapper
    return request_fun(self, *args, **kwargs)
  File "/opt/splunk/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/splunk/etc/apps/threatstream/bin/splunklib/binding.py", line 235, in _handle_auth_error
    raise AuthenticationError(msg, he)
splunklib.binding.AuthenticationError: Autologin succeeded, but there was an auth error on next request. Something is very wrong.

So I guess "Something is wrong"? but what?

Anyone knows a solution or at least the cause of this?

[end_es]

have you solved the issue?

[starcher]

I would open a support ticket with anomali. That’s their code. Something in way they are trying to hit kvstore.