How to automatically clean up stale python processes

Under certain circumstances we observe that the number of python processes (manage.py) serving the Columbus webapp will accumulate. Most of these processes end up in an unused state (only the 6 most recent processes should remain active) and can ultimately have a negative impact on the Columbus application.

The following command will provide a count of the manage.py processes on the Columbus server.

$ ps -fU columbus | grep [m]anage.py | wc -l

If the output is greater than 10, the unused processes can be safely killed using the following command:

$ kill `ps kstart_time -fU columbus | awk '$3 != 1 && /[m]anage.py/ {print $2}' | head -n -10`

This command lists the PID of all but the 10 most recently spawned manage.py processes owned by the columbus user, then passes those PIDs to the kill command for removal.

As it is safe to remove these processes whilst the Columbus application is running, the cleanup can be automated using cron. Create a file named 'columbus_python_cleanup' within the /etc/cron.daily directory on the Columbus server and give the file the following content:

~~~~~~~~~~~~~~~

#!/bin/bash -e

## script to clean up stale python processes

proc_limit=10
cmd="ps kstart_time -fU columbus | awk '\$3 != 1 && /[m]anage.py/ {print \$2}' | head -n -$proc_limit"

orphans=$(eval $cmd)

if [[ ! -z "$orphans" ]]
then
kill $orphans
fi

~~~~~~~~~~~~~~~

Now make the script executable.

$ chmod a+x /etc/cron.daily/columbus_python_cleanup

The script will be executed daily, but the cleanup command will only be executed if the number of python processes exceeds 10.