Cannot figure out where to find the number of threads that python is seeing

I am running into an issue that I do not fully understand.

My crontab runs python jobs on a schedule as it's supposed to. Recently they've started failing when loading the pandas library with the following error :

OpenBLAS blas_thread_init: pthread_create failed for thread 4 of 8: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 126940 current, 126940 max
OpenBLAS blas_thread_init: pthread_create failed for thread 5 of 8: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 126940 current, 126940 max
OpenBLAS blas_thread_init: pthread_create failed for thread 6 of 8: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 126940 current, 126940 max
OpenBLAS blas_thread_init: pthread_create failed for thread 7 of 8: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 126940 current, 126940 max

Even though I do not know why so many processes would be created by my jobs, this is still a fairly straightforward error.

So I tried to keep track of the number of open processes to see if there was a specific job triggering the increase. I did that using:

ps -eLo pid,cmd,nlwp | wc -l

which gives me a fairly stable number of around 5k even when the job is failing with the above message (so I was expecting to see ~126k processes).

I am at a loss here : which command do I need to run to be able to keep track of the number of processes that Python sees (i.e ~126940).

2 Sorted by: Reset to default

Know someone who can answer? Share a link to this question via email, Twitter, or Facebook.

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy