Local, non distributed app install in clustered search head environment

[fatsug]

This may be a "dumb" question, but I'll just throw it out there while I try to work it out.

The Python for Scientific Computing (PSC) app is HUGE. We have a clustered environment and Search Heads (SH) receive configuration from our deployer.

During initial setup the maximum bundle size was increased to allow pushing the PSC app from the deployer to the SHs. While it worked we've noticed that any push after adding the PSC app to the deployer now takes around 2 minutes to complete, regardless of how small of a change even if no restart is needed.

I was hoping there was a way to install the PSC app locally in the search head cluster without going through the deployer. To option to "install from file" is not present in the web UI, assumably since we have a deployer for managing apps.

Removing the app from the deployer and consequentially the SH cluster I tried to unpack the app into the /opt/splunk/etc/apps folder but it is then removed/deleted automatically as the cluster is restarted. Presumably since it is not available on the deployment server.

So, how should we install and use PSC in a clustered environment? Is the only/correct way to to push the giant app from the deployer or is there another way to distribute the app? 

All feedback and/or suggestions are welcome

[livehybrid · Accepted solution]

Hi @fatsug 

Unfortunately the only supported deployment process for this with a SHC is via the deployer as you have described. It isnt uncommon to need to increase the maximum bundle size, however it sounds like you've already dealt with that side of it.

Even though it is large, I dont think it should take quite so long to package and distribute - Its worth checking for any errors or warnings in _internal relating to this to see if there are any other underlying issues that could be slowing it down.

The other thing to check is if the PSC app directory on your SHD (e.g Splunk_SA_Scientific_Python_linux_x86_64) has previous copies of the app within it - e.g. have you extracted a newer version over the old? If so there may be a bunch of libraries/dependencies and other files from the old version which are no longer needed. If this is the case I would recommend backing this up and then deleting before re-extracting the latest version into your $SPLUNK_HOME/etc/shcluster/apps folder.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[fatsug]

Well crud...

The PSC os something like 2.3 GB, so even if the entire app is not redistributed every push I suspect it severly affects the check and push process.

No, it is a fresh and first time install so no inherited problems. As far as I can see, there are no errors or problems associated with the install itself and MLTK is running smoothly.

So this is another "improvement" to wish for then, something like a ".gitignore" option where you can manually add apps to your cluster without having them "managed" by the SHD.

My guess is that the only way around this would be a standalone SH where you'd basically only install PSC and MLTK and only use it for these purposes.

In any case, thank you for your feedback and have a nice weekend

[fatsug]

https://ideas.splunk.com/ideas/APPSID-I-989

[livehybrid]

Hi @fatsug 

Since the previous message I've had an idea - I havent been able to check it yet but if you added a local/app.conf inside the PSC app with the following:

[shclustering]
deployer_push_mode = local_only

Then I think it would push only local content from the PSC app on the deployer, Im assuming this would exclude the large bin directory out of the bundle?

Might be worth a try!

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[fatsug]

I am tempted to switch this post to the "solution", it is not "exactly" what I asked for but it did achiecve the effect I was looking for.

As per Use the deployer to distribute apps and configuration updates - Splunk Documentation. After an initial push of PSC and MLTK from the deployment server we added a local/app.conf file containing only 

[shclustering]
deployer_push_mode = local_only

The "push-time" dropped from 167 seconds to 47 seconds with push mode set in the PSC app, then to 24 seconds when also changing push mode in the MLTK app.

So this did have the effect I was after, even though it is not strictly speaking a "local only install" of an app.

All the best

[fatsug]

Did not know that you could do this on an "app level", but it might be worth looking into different push modes!

Thanx