Mutlisite SHC

[rahulhari88]

Hi 

I have a following architecture and i am trying to setup my Search head cluster .

i have multiple questions , 
if i want to have 1 copy of search artifact in each SH what should be my replication factor here in this command 

splunk init shcluster-config -auth <username>:<password> -mgmt_uri <URI>:<management_port> -replication_port <replication_port> -replication_factor <n> -conf_deploy_fetch_url <URL>:<management_port> -secret <security_key> -shcluster_label <label>

Second question is how will i set up the captain in this cluster 
If i run this command on the SH1 will it become captain 
opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list "https://splunk-essh01.abc.local:8089,https://splunk-essh02.abc.local:8089,https://splunk-essh03.abc...."

Last question is if i want to connect this SHC to the indexer cluster , will this command work 

splunk edit cluster-config -mode searchhead -site site0 -manager_uri https:// LB-IP-OR-DNS-HOSTNAME:8089   -replication_port 9887 -secret "<redacted>"

[livehybrid]

Hi @rahulhari88 

The docs on "Configure multi-cluster search for multisite indexer clusters" is also worth a read to understand how this is configured. There are also conf-file based examples as well as CLI examples incase you plan to commit your changes as config files in any repo/deployment system etc.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[livehybrid]

Hi @rahulhari88

To have 1 copy of each search artifact on every search head in your cluster, set the -replication_factor equal to the total number of search heads in your cluster. For example, if you have 3 search heads, use -replication_factor 3.

Example:

splunk init shcluster-config -auth <useername>:<password> -mgmt_uri <yourHost>:8089 -replication_port  -replication_factor 3 -conf_deploy_fetch_url <url>:<port> -secret <secret> -shcluster_label <yourLabel>

To bootstrap the captain, run the splunk bootstrap shcluster-captain command on any one search head (e.g., SH1). The node where you run this command will initially become the captain, but captaincy can change automatically later as this is dynamic (unless otherwise set as static).

Example:

/opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list "https://splunk-essh01.abc.local:8089,https://splunk-essh02.abc.local:8089,https://splunk-essh03.abc.local:8089"

To connect your search head cluster to the indexer cluster, your command is almost correct but you need to specify the site number.

Example:

splunk edit cluster-config -mode searchhead -site site<n> -manager_uri https://yourCMAddress:8089 -replication_port 9887 -secret ""

• The replication_factor determines how many copies of each search artifact exist in the cluster. Setting it to the number of search heads ensures every SH has a copy.
• Bootstrapping the captain on any member will make it the initial captain; captaincy may change due to elections.
• The edit cluster-config command with -mode searchhead is the correct way to connect your SHC to an indexer cluster.

The following docs might also be 

• https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/SHCdeploymentoverview
• https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Multisitearchitecture

🌟Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[isoutamo]

You could also use site0 as site information instead of site1 or site2. Then it manages searches little bit differently than using exact site<#>. You found more information from those docs which are pointed to you.

[rahulhari88]

@livehybrid : the reason for asking 2nd and 3 rd question is in case of 2nd i keep hearing that there needs to be 3 SH in a site in a cluster otherwise captain selection will be difficult , however i can see this architecture as well (https://www.splunk.com/en_us/pdfs/white-paper/splunk-validated-architectures.pdf) ,so is my config correct in that case .

for the 3rd question i have mentioned site as site0 indicating to disable site affinity, is that not correct should i still mention which site it is .

[kiran_panchavat]

@rahulhari88 

If this information meets your requirements, please proceed to accept the solution

[kiran_panchavat]

@rahulhari88 

You configure the site replication factor with the site_replication_factor.

site_replication_factor = origin:<n>, [site1:<n>,] [site2:<n>,] ..., total:<n>

where:
<n> is a positive integer indicating the number of copies of a bucket.

origin:<n> specifies the minimum number of copies of a bucket that will be held on the site originating the data in that bucket (that is, the site where the data first entered the cluster). When a site is originating the data, it is known as the "origin" site.

site1:<n>, site2:<n>, ..., indicates the minimum number of copies that will be held at each specified site. The identifiers "site1", "site2", and so on, are the same as the site attribute values specified on the peer nodes.

total:<n> specifies the total number of copies of each bucket, across all sites in the cluster.

You configure the site search factor with the site_search_factor

site_search_factor = origin:<n>, [site1:<n>,] [site2:<n>,] ..., total:<n>

where:
<n> is a positive integer indicating the number of searchable copies of a bucket.

origin:<n> specifies the minimum number of searchable copies of a bucket that will be held on the site originating the data in that bucket (that is, the site where the data first entered the cluster). When a site is originating the data, it is known as the "origin" site.

site1:<n>, site2:<n>, ..., indicates the minimum number of searchable copies that will be held at each specified site. The identifiers "site1", "site2", and so on, are the same as the site attribute values specified on the peer nodes.

total:<n> specifies the total number of searchable copies of each bucket, across all sites in the cluster.

[kiran_panchavat]

@rahulhari88 

Configuring the Multisite Manager Node

splunk edit cluster-config -mode manager -multisite true -site site1 -available_sites site1,site2 -site_replication_factor origin:1,total:2 -site_search_factor origin:1,total:2 -secret mycluster

Configuring Multisite Cluster Peer Nodes

Peer 1&2

splunk edit cluster-config -master_uri https://x.x.x.x:8089 -mode peer -site site1 -replication_port 9100 -secret mycluster

Peer 3&4

splunk edit cluster-config -master_uri https://x.x.x.x:8089 -mode peer -site site2 -replication_port 9100 -secret mycluster

Configuring a New Multisite Search Head

./splunk edit cluster-config -mode searchhead -master_uri https://x.x.x.x:8089 -site site2 -secret mycluster

Assign one of the members as the captain and set a member list:

./splunk bootstrap shcluster-captain –servers_list https://SH2:8089,https://SH3:8089,https://SH4:8089

[kiran_panchavat]

@rahulhari88 

Perform post-deployment set-up

Integrate the search head cluster with an indexer cluster (Single site)

./splunk edit cluster-config -mode searchhead -master_uri https://x.x.x.x:8089 -secret <secretkey>
./splunk restart

[rahulhari88]

@kiran_panchavat  : its a multi site environment not a single site . 

[rahulhari88]

@kiran_panchavat : Thanks for answering the first question , can you also check and provide your inputs on the 2nd and 3rd question as well .

[kiran_panchavat]

@rahulhari88 

Multisite clusters differ from single-site clusters in these key respects:

1. Each node (master/peer/search head) has an assigned site.
2. Replication of bucket copies occurs with site-awareness.
3. Search heads distribute their searches across local peers only, when possible.
4. Bucket-fixing activities respect site boundaries when applicable

Multisite and single-site nodes share these characteristics:

1. Clusters have three types of nodes: master, peers, and search heads.
2. Each cluster has exactly one master node.
3. The cluster can have any number of peer nodes and search heads.

Multisite nodes differ in these ways:

Every node belongs to a specific site. Physical location typically determines a site. That is, if you want your cluster to span servers in Bangalore and Hyderabad, you assign all nodes in Bangalore  to site1 and all nodes in Hyderabad to site2.

A typical multisite cluster has search heads on each site. This is necessary for search affinity, which increases search efficiency by allowing a search head to access all data locally.

[kiran_panchavat]

@rahulhari88 

Key points

site_replication_factor: controls how to distribute raw copies of data among the sites.
site_search_factor: controls how to distribute searchable copies

availabe_sites : defines the site in cluster.

site: A logical group that shares clustering policies, also the site where the master node resides.

multisite: Enables multi site clustering

[kiran_panchavat]

@rahulhari88 

Check these documentations:

https://docs.splunk.com/Documentation/Splunk/9.4.1/DistSearch/DeploymultisiteSHC 
https://docs.splunk.com/Documentation/Splunk/9.4.1/Indexer/Multisitearchitecture 

[kiran_panchavat]

@rahulhari88 

The replication_factor determines how many copies of search artifacts (e.g., knowledge objects, search results) are maintained across the search head cluster.

Choose the replication factor for the search head cluster - Splunk Documentation

For example: If you have 3 search heads (SH1, SH2, SH3), set replication_factor to 3. This ensures that every search head maintains a copy of the artifacts.

The command would look like:

/opt/splunk/bin/splunk init shcluster-config -auth admin:<password> -mgmt_uri https://<SH1>:8089 -replication_port 9887 -replication_factor 3 -conf_deploy_fetch_url https://<deployer>:8089 -secret <security_key> -shcluster_label shc1

Captain:

The captain is the search head that coordinates activities in the SHC, such as scheduling searches and replicating artifacts. Running this command on SH1 will designate SH1 as the initial captain. The captain role is dynamic and can move to another search head if the current captain fails, as long as the cluster has a (majority of nodes available).