Moving Indexers to a New Site with RF Adjustments

[m_zandinia]

Hi everyone,

I have 3 indexers (in a cluster) located on Site A. The current replication factor (RF) is set to 3.
I need to move operations to Site B. However, for technical reasons, I cannot physically move the existing data — I will simply have 3 new indexers at Site B.

Here’s the plan I’m considering:

1. Launch 1 new indexer at Site B.

2. Add the new indexer to the existing cluster.

3. Increase the RF to 4 (so that all raw data is fully replicated across the available indexers).

4. Shut down all 3 indexers at Site A.

5. Decrease the RF back to 3. (I understand there is a risk of some data loss.)

6. Add 2 additional new indexers to the cluster at Site B.

My main concern is step 5 — decreasing the RF — which I know is not best practice, but given my situation, I don't have many options.

Has anyone encountered a similar situation before?
I'd appreciate any advice, lessons learned, or other options I might not be aware of.

Thanks in advance!

[livehybrid · Accepted solution]

If you really dont want to go down the multisite route then you could keep your RF at 3 and slowly introduce new indexers in the new location, offlining one from the old site as each new one is added, although I really would recommend the multisite approach personally...

Here is what you would do:

1. Add all 3 new indexers to Site B while keeping Site A indexers active
2. Wait for full data replication to new indexers (verify with)
3. Gracefully decommission Site A indexers one at a time, waiting full rebalancing of buckets before doing the next one - splunk offline -auth <admin>:<password>

1. Cluster automatically rebalances data to maintain RF=3 during decommissioning

Why This Works as an approach

• Maintains RF=3 compliance throughout
• Avoids dangerous RF reduction step
• Uses Splunk's built-in rebalancing for safe peer removal

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[livehybrid]

Hi @m_zandinia 

I dont think upping your RF to 4 is the right approach here, you should probably treat this as a single-to-multisite cluster migration, even if you are going to deprecate the old site afterwards.

There are some useful docs at https://docs.splunk.com/Documentation/Splunk/9.4.1/Indexer/Migratetomultisite covering thism,

• Multisite clustering makes the cluster aware of physical locations (sites). You configure site-specific replication and search factors (site_replication_factor / site_search_factor).
• Add the new Site B indexers to the cluster, assigning them to a new site (site2).
• Configure the cluster manager for multisite operation, specifying that you need copies and/or searchable copies on both site1 (your existing Site A) and site2 (your new Site B).
• Configure the manager to convert legacy buckets to multisite
• The cluster manager automatically replicates data between sites to satisfy these policies, ensuring Site B receives a complete copy of the data over time.
• Once Site B has a full copy and the cluster is stable, you can safely decommission Site A indexers by updating the site policies, putting Site A peers in detention, waiting for buckets to be fixed, and then removing them.

Directly decreasing the Replication Factor (RF) when indexers holding necessary copies are offline can lead to data loss because the cluster manager may still believe those hosts exist.

Migrating data between sites using multisite replication takes time and network bandwidth. Monitor the cluster status closely (`splunk show cluster-status --verbose` or the Monitoring Console) to ensure replication completes before decommissioning the old site.

Plan your site_replication_factor and site_search_factor carefully based on your desired redundancy and search availability during and after the migration.

Useful Documentation Links:

• Multisite indexer clusters
• Decommission a site

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing