Execution Server High CPU

 We are on Automate Enterprise 11.0.17 and over a 10 day period we've noticed CPU utilization slowly increase to 50pct, after 20 days it is in the 90pct range. We have around 60 workflows. The most frequent of which trigger ever 15 minutes.Is there any way to prevent this, as it eventually starts to degrade performance? 

Hello Barnes,

Are all components installed on the same machine? If not can you describe your setup?

Are there any errors in the windows Event Viewer (under application)?

What types of workflows are you running (SQL, interactive, background)?

Yes management, execution, and agent all run on the same machine (Windows 2012 R2 Standard).

We do see errors from time to time with one workflow, it downloads files from an SFTP that sometimes contain invalid characters and it causes "AML is Invalid" errors in the event log.

Most of our workflows are triggered by schedule conditions (we have 60 or so conditions and workflows), we do use several file watchers. Most tasks invoked by the workflows move files, invoke other applications or scripts, and a few query SQL datasources. 

Hello Barnes,

Which process is taking the CPU when the issue occurs? Is it BPA_Man.exe? BPA_Exec.exe?

Do you have any errors regarding workflows getting stuck or not executing properly?

You mention you had tasks that use FTP actions and SQL queries. When opening the FTP connection, do you have a "close connection" step? Same for the SQL connections?

BPA_Exec.exe is the task that ends up consuming the CPU. After a reboot though it will remain in low usage for days or even weeks. Other than the error mentioned above, we don't typically see too many (unless a remote SFTP site is down etc). We typically close FTP / SQL connections. 

Hi Barnes,

First and foremost, I recommend updating to the latest version of Automate Enterprise. A lot of corrections have been applied since your version. I have reviewed your support cases and I do not see a case referencing a similar issue. I recommend, on top of updating, to open a support case so that we can schedule a remote session with you to further troubleshoot. 

-Leonard

Hi Barnes,

I have been working with Automate support on this same type issue as well. The latest release 11.1.10.5 was to address the BPA_MAN.exe memory "leak" and it appeared to do so. Unfortunately the release was unstable in our environment and I had to revert back to 11.1.1. I still had the issue with the BPA_EXEC.exe "leaking" memory though. I have support tickets open on both items. Thought I would share :)

We see both elevated memory and cpu usage. It caused the server to become unresponsive over the weekend. From what I can tell it appears to occur when a user is logged into management studio for extended periods of time. When we see it typically a user will RDP to the server, login to the management tool, disconnect from RDP without signing out, and leave the tool open for several hours or more.

Hi Barnes,

We have had some customers report elevated memory usage in the Management Service process when the SMC is left open and connected for an extended period of time. Depending on your license, you may have Access to the Management Console Sessions feature under the Options/Server Settings page. Here you can enable session time out by users. The timeout is for inactivity. This should help with the situation you are describing. We still would recommend you update to the 11.1.10.5 release to see if the changes made provide relief for the High CPU usage issues.

As an update I tried to update to 11.1.10.5 and it seems to have broken SSL/TLS, which is a requirement for us. I submitted a ticket and they created a bug case (#24736). 

Has there been any progress on this? We still aren't able to update due to bug # 24736 (SSL not working in latest version). 

Hello Barnes,

I do apologize however this issue is still being worked on by our development team. This thread will be updated once the issue has been corrected. 

As of updating to 11.1.20.19 we are no longer seeing CPU usage "creep" or elevated cpu usage.

how about the memory leak issue?  I have to restart my automate serves daily or my server will become unresponsive and has to be restarted.

Which version are you on? On 11.0.1.17 we would get up to 1500MB range before our weekly reboot. Now it seems to sit around 350-450. We are only rebooting monthly at the moment. Out of curiosity, approximately how many workflows are you running? We have about 90 workflows with their own schedule conditions.