OK. Let's start at the start 🙂 index=finder_db AND (host="host1" OR host="host2") AND (("Wonder Exist here") OR ("Message=Limit the occurrence" AND "FinderField=ZEOUS")) This will select the events for further processing. But the question is whether you're extracting any fields from those events. Before we're going anywhere further, we need to know whether: 1) The uniqueId field (to which you're referring in subsequent posts in a case-inconsistent manner) is extracted. 2) The "data" field(s) which you want to "merge" are extracted. Generally, the field extraction should be (actually, should already have been) handled at data onboarding stage. When you have this one covered, you can get to the second part - handling the logic behind "joining" your events. ... View more

I assumed your fields are already extracted. After some thought, actually the stats doesn't add anything here. It should be enough to just do the xyseries. As long as you have fields properly extracted. ... View more

I'm not sure what "columns" you want from this data but assuming that you want to have a table with various messages per id and status you might want something like <your initial search> | stats values(Message) as Message by UniqueId Status | xyseries UniqueId Status Message ... View more

No. They are not being broken. The SEDCMDs are being applied and apparently it removes part of the event data so the remaining data sometimes happens to be valid JSON and sometimes isn't. But it has nothing to do with event breaking. ... View more

If I remember correctly, the alert created "normally" with a "create notable" action should work as well (although it will have less configuration options). But in order to be able to create notables the user who the alert is run with must have proper privileges within the ES app. So if an alert was created with a normal alert creation means instead of ES Content Management, the user might not be able to create notable due to insufficient permissions. (yes, I'm aware that it sounds a bit convoluted). ... View more

The CLONE_SOURCETYPE option in a transform causes Splunk to create a copy (at this moment of the ingestion pipeline, so all the state of the event at this point is retained) of the processed event, changes its sourcetype to the one specified in the CLONE_SOURCETYPE option and reingests the event back at the (almost) beginning of the pipeline (skipping the initial phases of line breaking and time recognition). See the usual https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platform-the-Masa/m-p/590774 CLONE_SOURCETYPE is called from a transform in the typing phase. The original event is processed without changes as if the transform containing the CLONE_SOURCETYPE option wasn't there. But the copy is moved back to the typing queue and starts the whole typing phase with a new sourcetype and triggers completely new set of transforms according to the new sourcetype (and possibly new source and host if they were overwritten during the initial transforms run).   ... View more

Apparently you have problems with proper extraction of the message field. So you should verify your data onboarding and start with fixing the extractions. ... View more

It's hard to say what's "wrong" not knowing your data but while transaction can be sometimes useful (in some strange use cases) it's often easier, and faster to simply use stats. Mostly because transaction has loads of limitations that stats don't have. Quick glance at your search suggests that for some reason the message field is not extracted properly from your event so you're not getting two separate values in your multivalued message output field. As I said I'd go with index=... ("SENDER[" OR ("RECEIVER[" AND "POST /my-end-point*")) | rex "\[(?<id>\d+)\]" | eval request=if(searchmatch("SENDER[",message,null()) | eval response=if(searchmatch("\"RECEIVER[\" AND \"POST /my-end-point*\"",message,null()) | stats range(_time) as duration, count, values(request) as request, values(response) as response, values(_raw) as _raw by id   ... View more

Ok, a word of advise - it's usually better to specify indexes explicitly than to have them as searched by default. Especially with an admkn role! It spares you unnecessary load on your environment for searches in which you haven't specified the indexes and it saves you a lot of debugging when you have different roles with different default indexes and people report mismatch in searches functionality. You have been warned. One additional hint - it's way better to do a quick check with | tstats count where index=rapid7 by sourcetype than index=rapid7 | stats count by sourcetype The first one only checks the summarized indexed fields while yours needs to plow through all events from the index. And there is something that doesn't add up. On Cloud you cannot have the admin user role. You can only have sc_admin (which is a limited admin role). So if you're trying to edit the admin role you shouldn't be able to do so.   ... View more

Firstly, let me start by stating the obvious - vulnerability scanners are notorious for being way overly trigger-happy with their findings. It takes an experienced person to filter their results and get the actual reasonable results. Having said that - those processes are spawned by the splunkd process (not directly -  via compsup daemon). So that finding is at least questionable if not simply a false positive. ... View more

What exactly do you want to do? The command you provided will "empty" the index without touching its definition. Also, I haven't tried this in a cluster (I assume that's what you mean by 3 indexers and "a management node") but I'd expect the cluster to start fixups as soon as you do the operation on the first node unless you enable maintenance mode. Anyway, if you want to leave the index definition but only remove the indexed events, that's one of the possibilities. Another one is to set very short retention period and let Splunk roll the buckets normally. If you want to remove the index along with its definition, you have to remove it from indexes.conf on the CM, push the config bundle (this will trigger rolling restart of indexers) and then manually remove index directories from each indexer. ... View more

Wait. As far as I remember (it's been some time since I did it last time) you don't manually copy anything. When you run the installer in deployer mode it takes care of preparing the shcluster bundle. That's why you run it exactly as described - upload the app to the deployer, run the installer on the deployer, apply shcluster-bundle. No manual copying stuff anywhere. ... View more

That's more or less what I was talking about 🙂 ... View more

I don't recall ever seeing dbconnect configured so that it sends to a HEC input outside of the HF it's running on. Theoretically it's possible - see https://docs.splunk.com/Documentation/DBX/3.18.2/DeployDBX/settingsconfspec but I must say I've never seen it configured this way. Anyway, first check your config., then debug apropriate HEC inputs. ... View more

Be also aware of the subsearch limitations. You can't run them for long and they have a limit for returned results. So if you run them over a big data set you might get incomplete results and since the subsearches will get silently finalized you won't even know about it. So i  your case it would probably be better to search for all matching events initially and tag them according to matching specific set of conditions using conditional field assignment ( | eval something=if( [...] )) ... View more

I don't understand your objections vs. methods 1 and 2. Complexity of the json structure shouldn't matter as long as the event is a valid json and - in case 2 - doesn't exceed maximum number of fields handled by auto-kv. ... View more

If you do tstats by time without binning and then do bin, you'll have to stats again to summarise your data. Bin on its own doesn't aggregate data, just aligns the field into discrete points. ... View more

There is one more thing I missed. You said that you're receiving the events via HEC input. Which endpoint are you using? I'd have to check about the /raw endpoint but the /event endpoint bypasses line breaking completely. So regardless of whatever you set as line breaker, the events are coming in as they are received. ... View more

1. Again - where did you put the fields.conf? (but this shouldn't affect tstats) 2. Do you have any other _meta definitions on your UF. Did you verify the effective config with btool? 3. Try  | walklex index=index_abc type=field over a longer time span and see if you get your id  as one of the results.   ... View more

Adding to valid @livehybrid points, you should set INDEXED_VALUE=false. It has nothing to do with the issue at hand but without it you won't be able to search for id=123 if then"123" string isn't contained within the raw event. ... View more

While I fully agree with @yuanliu and @bowesmana that map is most probably not the way to go in this case there is an obvious lack of/improper use of quotes. If you unescape your mapped search you'll get search index=xyz "$traceid$" AND "REQUEST BODY" | rex field=msg "artifact_guid":"(?<artifact_guid>[a-f0-9-]{36})" | rex field=msg "email_address":"(?<email_address>[^"]+)" | table traceid, artifact_guid, email_address This is not the right syntax for rex command. As far as I remember map won't pass the errors from the mapped search to your outer search - it will just not yield any results. Hence your behaviour. ... View more

1. Line breaking is happening before sedcmd kicks in so you must adjust your line breaker to work on a "non-trimmed" data and then remove the header/footer from the broken events. 2. Your line breaker doesn't account for new line characters. 3. You are aware that if you simply break at curly braces you can't have any more levels in your json structure because it will break your breaking, right? So you should first break at ([\r\n\s]+){ Then you can use sedcmd to remove the dangling [\s\n\r]*\][\s\n\r\]*} part. And use a transform to drop (send to nullqueue) the header event. ... View more

TLS can be sensitive to errors and it's relatively easy to break your environment so I'd advise you to do some lab work (not necessarily with Splunk; apache httpd for example is a very well docummented solution you can use to get experience with real-world third-party CA - you can use Let's Encrypt for free). But back to your question - as a rule of thumb, the side which identity you need to verify must use a certificate trusted by the other side. So typically - like with most of the HTTPS servers over the internet - the server presents a certificate issued by a CA that the client trusts. That's the basic minimal setup which allows the client to be sure that the server is who it claims to be and then allows it to negotiate a protected encrypted connection within which it can, for example, authenticate itself using normal HTTP means. This is the way it works normally between your browser and most internet services - your browser has a list of trusted RootCAs. The server you're trying to connect to presents a certificate with certificate chain tracing back to a RootCA your browser trusts. The browser checks if the certificate is valid, matches the server you wanted to connect to and the purpose for which the certificate was issued (so you can't just set up a www server with a certificate meant for S/MIME mail encryption). Your browser negotiates an encrypted connection with the server and within this connection, since you can now trust that the server is who it claims to be, you authenticate yourself using login/password or whatever other method you use because you know that you're talking to a known party over a secure channel. That's a typical use case and it matches the typical use case with UF as a client (both as a client for DS as well as a client connecting to the indexer/HF). You need to have a certificate issued by a trusted CA (or having a certification chain back to trusted CA) on the server (DS, idx and so on) and have the UF trust the CA certificate. In this case all UFs just need to have this one CA certificate configured as a trusted CA. But you can also if needed configure client's authentication. In such case each UF would have its own certificate which in this case would need to be issued by a CA trusted by the server. In this scenario, typically referred to as mTLS or mutual authentication, each side verifies the identity of the other endpoint so each unique subject (in our case each UF) would require a separate individual cert. This scenario is however difficult to manage without a lot of administrative overhead (just tracking expiration dates of dozens of certs is enough of a headache, not mentioning renewing them and installing) so it's not commonly used. But it's possible. ... View more

First tag your event with one of two classes | eval type=if(isnull(contactid),"lw_message","standardised_message") And then do | xyseries id type time ... View more