Fully featured low overhead profiler for Java EE and Java SE platforms.
Ultimate profiling and monitoring solution for Gradle, Maven, Ant, JUnit and TestNG.
Easy to use performance and memory profiler for .NET framework.

Too Many Open files issue

Moderators: Vladimir Kondratyev, Anton Katilin

Too Many Open files issue

Postby sidchhabra » Wed Aug 12, 2015 10:32 pm

Hi
We are getting this error after running our application under some load for 2-3 hours.
We attached yourkit profiler to it and saw the following
After 50 minutes
FileInputStream Opened 34 Closed 31
FileOutputstream Opened 42 Closed 37
Sockets Opened 1871 Closed 1783

-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep [i]1/conf | awk '{print $2}'`/fd | wc -l | xargs echo "fd"
fd 3188
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep [i]1/conf | awk '{print $2}'`/fd | grep pipe | wc -l
1900
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep [i]1/conf | awk '{print $2}'`/fd | grep eventpoll | wc -l
950
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep [i]1/conf | awk '{print $2}'`/fd | grep "/" | wc -l
219
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep [i]1/conf | awk '{print $2}'`/fd | grep socket | wc -l
118

So I had the following questions:
1. whats the differenrce between pipe and eventpoll and why do these keep increasing ?
2. Sockets open closed is not sufficient to debug this further as the currently opened sockets are not high enough
3. How do I debug this further ?
sidchhabra
 
Posts: 4
Joined: Wed Aug 12, 2015 9:40 pm

Re: Too Many Open files issue

Postby sidchhabra » Wed Aug 12, 2015 10:34 pm

We are using the following
-bash-3.2$ java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
-bash-3.2$ cat /etc/*-release
CentOS release 5.5 (Final)
sidchhabra
 
Posts: 4
Joined: Wed Aug 12, 2015 9:40 pm

Re: Too Many Open files issue

Postby sidchhabra » Wed Aug 12, 2015 11:18 pm

After 2 hours
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep [i]1/conf | awk '{print $2}'`/fd | wc -l | xargs echo "fd"
fd 7744
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep [i]1/conf | awk '{print $2}'`/fd | grep -c pipe
4938
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep [i]1/conf | awk '{print $2}'`/fd | grep -c eventpoll
2469
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep [i]1/conf | awk '{print $2}'`/fd | grep -c socket
87
-bash-3.2$ sudo -u matchmaker ls -l /proc/`ps -ef | grep [i]1/conf | awk '{print $2}'`/fd | grep -c "/"
219

And the limit is 8193 for fd so within a few minutes we will run out of file descriptors
sidchhabra
 
Posts: 4
Joined: Wed Aug 12, 2015 9:40 pm

Re: Too Many Open files issue

Postby Anton Katilin » Mon Aug 17, 2015 1:15 pm

Hello,

Could you please clarify: do you mean you think the descriptors are opened and closed by the profiler agent itself, and you did not see this problem when running your application without the profiler?

Or do you mean that you have noticed the growth by using the profiler?

Anyway, what version of the profiler do you use?

Did you use the attach mode or started the profiled application with the profiler agent?
https://www.yourkit.com/docs/java/help/attach_agent.jsp

Best regards,
Anton
Anton Katilin
 
Posts: 5576
Joined: Wed Aug 11, 2004 8:37 am

Re: Too Many Open files issue

Postby sidchhabra » Mon Aug 17, 2015 3:15 pm

No the descriptors are not opened by the profiler
They are opened by our application
And we are trying to find the piece of code causing this
We are using 2015_build_15072
We used the profiler agent
sidchhabra
 
Posts: 4
Joined: Wed Aug 12, 2015 9:40 pm

Re: Too Many Open files issue

Postby Anton Katilin » Mon Aug 17, 2015 3:33 pm

For this purpose please use the Events:
https://www.yourkit.com/docs/java/help/probes_ui.jsp

The probes are in Auto state by default,
https://www.yourkit.com/docs/java/help/ ... probes.jsp

...so the easiest way to start getting the results is to start CPU sampling thus enabling the probes:
https://www.yourkit.com/docs/java/help/cpu_intro.jsp
Anton Katilin
 
Posts: 5576
Joined: Wed Aug 11, 2004 8:37 am


Return to Java Profiler

Who is online

Users browsing this forum: youthforchange and 5 guests