Too Many Open files issue

Questions about YourKit Java Profiler
Post Reply
sidchhabra
Posts: 4
Joined: Wed Aug 12, 2015 9:40 pm

Too Many Open files issue

Post by sidchhabra »

Hi
We are getting this error after running our application under some load for 2-3 hours.
We attached yourkit profiler to it and saw the following
After 50 minutes
FileInputStream Opened 34 Closed 31
FileOutputstream Opened 42 Closed 37
Sockets Opened 1871 Closed 1783

-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep 1/conf | awk '{print $2}'`/fd | wc -l | xargs echo "fd"
fd 3188
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep 1/conf | awk '{print $2}'`/fd | grep pipe | wc -l
1900
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep 1/conf | awk '{print $2}'`/fd | grep eventpoll | wc -l
950
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep 1/conf | awk '{print $2}'`/fd | grep "/" | wc -l
219
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep 1/conf | awk '{print $2}'`/fd | grep socket | wc -l
118

So I had the following questions:
1. whats the differenrce between pipe and eventpoll and why do these keep increasing ?
2. Sockets open closed is not sufficient to debug this further as the currently opened sockets are not high enough
3. How do I debug this further ?
sidchhabra
Posts: 4
Joined: Wed Aug 12, 2015 9:40 pm

Re: Too Many Open files issue

Post by sidchhabra »

We are using the following
-bash-3.2$ java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
-bash-3.2$ cat /etc/*-release
CentOS release 5.5 (Final)
sidchhabra
Posts: 4
Joined: Wed Aug 12, 2015 9:40 pm

Re: Too Many Open files issue

Post by sidchhabra »

After 2 hours
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep 1/conf | awk '{print $2}'`/fd | wc -l | xargs echo "fd"
fd 7744
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep 1/conf | awk '{print $2}'`/fd | grep -c pipe
4938
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep 1/conf | awk '{print $2}'`/fd | grep -c eventpoll
2469
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep 1/conf | awk '{print $2}'`/fd | grep -c socket
87
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep 1/conf | awk '{print $2}'`/fd | grep -c "/"
219

And the limit is 8193 for fd so within a few minutes we will run out of file descriptors
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Too Many Open files issue

Post by Anton Katilin »

Hello,

Could you please clarify: do you mean you think the descriptors are opened and closed by the profiler agent itself, and you did not see this problem when running your application without the profiler?

Or do you mean that you have noticed the growth by using the profiler?

Anyway, what version of the profiler do you use?

Did you use the attach mode or started the profiled application with the profiler agent?
https://www.yourkit.com/docs/java/help/attach_agent.jsp

Best regards,
Anton
sidchhabra
Posts: 4
Joined: Wed Aug 12, 2015 9:40 pm

Re: Too Many Open files issue

Post by sidchhabra »

No the descriptors are not opened by the profiler
They are opened by our application
And we are trying to find the piece of code causing this
We are using 2015_build_15072
We used the profiler agent
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Too Many Open files issue

Post by Anton Katilin »

For this purpose please use the Events:
https://www.yourkit.com/docs/java/help/probes_ui.jsp

The probes are in Auto state by default,
https://www.yourkit.com/docs/java/help/ ... probes.jsp

...so the easiest way to start getting the results is to start CPU sampling thus enabling the probes:
https://www.yourkit.com/docs/java/help/cpu_intro.jsp
Post Reply