About Nicolas2203

Nicolas2203

OK now I get it, so I need to define one more stanza in my transforms to overwrite the _TCP_ROUTING For example [mydevice_overwrite_tcprouting] REGEX = . DEST_KEY = _TCP_ROUTING FORMAT = an_empty_output Props.conf [my_device_clone] TRANSFORMS-deletetcprouting = mydevice_overwrite_tcprouting Something like this ? is it possible to define an empty values in an outputs.conf ? Thanks Nicolas

Nicolas2203

Ok so if I understand, when I clone a sourcetype, he will clone it's destination too ? Not sure to understand, I have some other log sources that I clone, and forward to a secondary splunk with the same clone methods. Cloning the sourcetype to sourcetype:anonymized In the transforms I applied on the cloned sourcetype some regex for anonymization And this sourcetype is routed via _TCP_ROUTING to an output that is a heavy forwarder that route to the secondary splunk. For example Transforms.conf [firewall_log-clone] CLONE_SOURCETYPE = firewall_log:clone REGEX = .* DEST_KEY = _TCP_ROUTING FORMAT = output_externalhf Props.conf [firewall_log] TRANSFORMS-clone = firewall_log-clone This is working, logs are properly sent to an HF that will forward those logs to a secondary splunk But it's not syslog log source so maybe this is the difference ? Thanks for the help

Nicolas2203

Hi @livehybrid Yes in my meening, setting a _TCP_ROUTING on the input will forward the logs on the local indexers. And, at the sametime, sourcetype is cloned to be forwarded to the second splunk. Maybe I didn't understand something 😕

Nicolas2203

Hi Splunk Community, I have a new issue concerning this config, some particular behaviour that I don't understand Here is my configuration #classic input on TCP, this is syslog logs [tcp://22000] sourcetype = mydevice:sourcetype index = local_index _TCP_ROUTING = local_indexers:9997 # Idea is to clone the sourcetype, but not logs containing LAN1 and LAN2 logs, it's not necessary for the second splunk [mydevice-clone] CLONE_SOURCETYPE = mydevice:clone REGEX = ^((?!LAN1|LAN2).)*$ DEST_KEY = _SYSLOG_ROUTING FORMAT = sending_to_second_splunk # on the props I apply the configuration made on the transforms [mydevice:sourcetype] TRANSFORMS-clone = mydevice-clone #IP of the HF that will send data to second splunk [syslog:sending_to_second_splunk] server = 10.10.10.10:port type = tcp Issue Encountered This configuration works partially: Data is properly indexed to the second Splunk, without LAN1 and LAN2 data. Data containing LAN1 and LAN2 is indexed on the local indexer. However, the sourcetype mydevice:clone is also indexed on my local indexer, resulting in some data being indexed twice with two different sourcetypes. I don't understand why this is happening and I am seeking help to resolve this issue, I have the feeling that I miss something Thanks, Nicolas

Nicolas2203

Thanks all for the help, I will try a regex that match both. I learn a lot with you guys thanks !!!!

Nicolas2203

Hello Had an additional questions, if I have to extract datas from two fields For example, I will NOT send to distant splunk, all the datas that in the dest_zone and src_zone contains DMZ-NETWORK and GUEST-NETWORK My config is something like that TRANSFORMS.CONF [clone_only_dmz-network] REGEX = ^((?!dmz-network).)*$ CLONE_SOURCETYPE = cloned:firewall_clone DEST_KEY = _SYSLOG_ROUTING #it's syslog logs FORMAT = distant_splunk PROPS.CONF [firewall_sourcetype] TRANSFORMS-clone = clone_only_dmz-network OUTPUTS.CONF [syslog:distant_splunk] server = ip of the distant HF This is actually working, I tested it But, if I want to not send mutliple fileds, how can I achieve that ?

Nicolas2203

Hi @livehybrid , @isoutamo Thanks a lot for your help! I’ll try the configuration with CLONE_SOURCETYPE and will come back here to let you know if it works for me. 🙂 Thanks again for your support!

Nicolas2203

Hello @livehybrid Thanks for your time OK, I understand now. I see what I was missing. Strangely, what I had done was working, and I was perplexed about that. 😕 I will test with the configuration you provided; it makes more sense. But I have a quick question: if the logs need to be anonymized before they are sent to the distant_HF, will putting the two outputs in the _TCP_ROUTING in the inputs.conf work? Many thanks for you clear answer !!!

Nicolas2203

Hello kiran, yes this is a typo .

Nicolas2203

Hi splunk community, I have a question on logs cloning/redirection Purpose : Extract logs containing "network-guest", and don't redirect this logs to a distant HF, but only to local indexers LOGS ENTRY CONFIG Into an app Splunk_TA_FIREWALL inputs.conf [tcp://22000] sourcetype = fw:firewall index = fw_index _TCP_ROUTING = local_indexers This logs are perfectly working and are stored on my local indexers Now this logs must be cloned and redirected to a distant HF but not the logs containing "network-guest" THat my props and transforms config props.conf [fw:firewall] TRANSFORMS-clone = fwfirewall-route-network-guest-, fwfirewall-clone transforms.conf [fwfirewall-route-network-guest] REGEX = \bNETWORK-GUEST\b DEST_KEY = _SYSLOG_ROUTING FORMAT = local_indexers [fwfirewalll-clone] DEST_KEY = _SYSLOG_ROUTING FORMAT = distant_HF REGEX = . When I check into the logs, on the distant splunk, I don't see NETWORK-GUEST logs anymore, and I can see those logs on the local splunk Question is, I'm not sure I'm doing that the right way, and not sure if it works 100% Has someone a good knowledge on this kind of configuration ? Thanks a lot for the help Nico

Nicolas2203 · ‎02-21-2025

Hello @livehybrid , yes, data are still getting in the original index and they contain "TheAppResourceGroupName"

Nicolas2203 · ‎02-21-2025

Hello, My use case : Context : On azure, datas from several applications are pushed in a Azure EventHub I need to separate the datas from one application, and put this datas into a new index on Splunk On Azure, all the resources of this app are in one Ressource Group : TheAppResourceGroupName I used a Heavy Forwarder, and this are my configs : props.conf : [source::eventhub://EVENTHUBAZURE.servicebus.windows.net/app-logs;] TRANSFORMS-route = routeToNewIndex, discard_original, transforms.conf [routeToNewIndex] REGEX = TheAppResourceGroupName DEST_KEY = _MetaData:Index FORMAT = NewIndex [discard_original] REGEX = TheAppResourceGroupName DEST_KEY = queue FORMAT = nullQueue This config will delete the datas, yes, but in the NewIndex, and not in the original Index, after the routing. I didn't find an answer witch fit with my needs on the commu and the docs, but maybe someone has to face a similar need . Thanks a lot for the help! Nico

Nicolas2203 · ‎10-17-2024

Hello, I just checked, and the Microsoft Cloud Services manage checkpoints locally on heavy forwarders. However, there is a configuration in the app that allows you to store checkpoints in a container within an Azure storage account. This way, when you need to start log collection on another heavy forwarder, it could facilitate the process. Will configure that and test, I let you know ! Thanks Nico

Nicolas2203 · ‎10-15-2024

Hello @PickleRick Thats an interesting topic, I will dig more information about it. I let you know here If I found something interesting Thanks ! Nicolas

Nicolas2203 · ‎10-14-2024

Hello @gcusello Thanks for the answer. Ok I understand, I will install the app on both HF and just activate it on one. When you say yu have to manage checkpoints between HFs. How is that possible in Splunk ? Assuming that logs are stored on the source for 2 weeks in case of an outage, when I activate log collection on the second HF, it will start collecting logs from the day it is activated, and it won't be aware of the logs already ingested into Splunk?

Nicolas2203 · ‎10-11-2024

Hi Splunk community, I have a quick question about an app, such as the Microsoft Cloud Services app, in a multiple Heavy Forwarder environment. The app is installed on one Heavy Forwarder and makes some API calls to Azure to retrieve data from an event hub and store this data in an indexer cluster. If the Heavy Forwarder where the add-on is installed goes down, no logs are retrieved from the event hub. So, what are the best practices for this kind of app, which retrieves logs through API calls, to be more resilient? The same applies to some Cisco add-ons that collect logs from Cisco devices via an API. For now, I will configure the app on another Heavy Forwarder without enabling data collection, but in case of failure, human intervention will be needed. I would be curious to know what solutions you implement for this kind of issue. Thanks Nicolas I'm curious

Nicolas2203 · ‎07-24-2024

The storage will on a DELL EMC storage, and considering the splunk recomandations and the SAN caracteristics, it will work smoothly, on the paper. I will test with few index and check Thanks a lot for your help !

Nicolas2203 · ‎07-24-2024

Ok I understand. And considering having two different NFS volumes on a SAN, one volume for each indexer, but the mounting point on the OS will have the same name for both indexers Can this solution work ?

Nicolas2203 · ‎07-23-2024

Hi marnall, thanks for the answer. So there is no chance to put cold datas on a NFS network storage without implementing smartstore ?

Nicolas2203 · ‎07-23-2024

Hello, Just have a question, about a test that I have in progress. I have one indexer cluster, with two servers. I decided to move, the colddb path, on a san share, mounted on both indexers. For on test index, I copy all the colddb buckest into the new space on the SAN, then, I change the configuration of my test index, push it on both indexers with the master node, and all seems to be OK. My question is about, how will splunk manage the colddb, it both indexers are pointing on the same share ? There is something that I'm not sure, splunk indexer will manage properly the fact, if indexer1 already save the bucket in the new colddb path ? or I will have buckets in double. Thanks for your clarifications

Nicolas2203 · ‎04-11-2024

Ok, to be honest I had to check more on the config to properly clone and forward my datas, the behaviour of the conf it's strange. But thanks a lot for your help, I appreciate !

Nicolas2203 · ‎04-09-2024

Ok I understand what you say. But sorry I forgot to mentionned that I have a TCPOUT default on my conf: [tcpout] defaultGroup = my_indexers forceTimebasedAutoLB = true forwardedindex.filter.disable = true [tcpout:my_indexers] server = indexer1:9997, indexer2:9997 So if I'am correct, the inputs.conf : [inputs.conf] [udp://22210] index = my_logs_indexer sourcetype = log_sourcetype disabled = false Redirect the logs to the default outputs, because no outputs is specified . Correct me if I'm wrong, and sorry to forgot this config at the first question

Nicolas2203 · ‎04-09-2024

Hi PickleRick, thanks for your response and time. The cloned logs are routing only to one instance, specified into the outputs.conf The "original" logs, not the cloned one are directed to my local indexers, just the cloned sourcetype is directed with another heavy forwarder specified in the outputs.conf placed in the same app as the props and transforms. Not sure if i'm clear 😕

Nicolas2203 · ‎04-08-2024

Hi all After temptative for troubleshooting my issue alone, I will try my luck here. Purpose : clone one sourcetype to store the logs into a local indexer, and in a distant one I use one heavy forwarder to receive the logs, store the logs in a indexer, and same heavy forwarder will clone the sourcetype to forward the cloned one into a distant heavy forward, that I don't managed. Here is my config : [inputs.conf] [udp://22210] index = my_logs_indexer sourcetype = log_sourcetype disabled = false This works pretty well, and all logs are stored into my indexer Now will come the cloning part : [props.conf] [log_sourcetype] TRANSFORMS-log_sourcetype-clone = log_sourcetype-clone [transforms.conf] [log_sourcetype-clone] REGEX = . DEST_KEY = _TCP_ROUTING FORMAT = distant_HF_formylogs [outputs.conf] => for cloned logs [tcpout:distant_HF_formylogs] server = ip_of_distant_HF:port sendCookedData = false This configuration is used for another use case, as sometimes I have had to anonymize some logs. However, for this particular use case, when I activate the cloning part, it stops the complete log flow for this use case, even on the local indexers. I didn't quite understand why, because I don't see the difference with my other use case, apart from the fact that the logs are UDP logs and not TCP. Am I missing something? Thanks a lot for your help

Nicolas2203 · ‎02-15-2024

OK thanks richgalloway and isoutamo for the help So if I understand correctly, if no outputs.conf is defined in the app conf then : - Splunk will used the first outputs file in the btool list, unless a default group is set ?

Posts	27
Solutions	0
Karma Given	10
Karma Received	0
Member Since	‎10-05-2023

Online Status	Offline
Date Last Visited	Monday

Redirecting specific logs using a regex

Deleting datas on one index after routing them to ...

Splunk multiple heavy forwarders and Splunk Add On...

indexer cluster : colddb moved on a san share

Sourcetype cloning - logs stopped

Noob question about btool and conf file precedence

Re: Redirecting specific logs using a regex

Re: Redirecting specific logs using a regex

Re: Redirecting specific logs using a regex

Re: Redirecting specific logs using a regex

Re: Redirecting specific logs using a regex

Re: Redirecting specific logs using a regex

Re: Redirecting specific logs using a regex

Re: Redirecting specific logs using a regex

Re: Redirecting specific logs using a regex

Redirecting specific logs using a regex

Re: Deleting datas on one index after routing them...

Deleting datas on one index after routing them to ...

Re: Splunk multiple heavy forwarders and Splunk Ad...

Re: Splunk multiple heavy forwarders and Splunk Ad...

Re: Splunk multiple heavy forwarders and Splunk Ad...

Splunk multiple heavy forwarders and Splunk Add On...

Re: indexer cluster : colddb moved on a san share

Re: indexer cluster : colddb moved on a san share

Re: indexer cluster : colddb moved on a san share

indexer cluster : colddb moved on a san share

Re: Sourcetype cloning - logs stopped

Re: Sourcetype cloning - logs stopped

Re: Sourcetype cloning - logs stopped

Sourcetype cloning - logs stopped

Re: Noob question about btool and conf file preced...

Are you a member of the Splunk Community?