The cause and possible solutions depend on the version of JVM you are using to profile your Java applications.
When a memory snapshot is being captured, objects in the heap are temporarily "tagged" with 8-byte numbers. JVMTI, the profiling API of Java 5 and newer, provides all heap data in terms of these tags. The mapping is stored inside JVM in some JVM internal data structures. When JVM fails to allocate memory for these structures, it terminates with an error.
A profiler agent doesn't have the means to reliably predict the lack of memory to prevent the JVM crash.
This problem is mostly a problem of 32-bit JVMs, that have limited virtual memory available (2-4 GB, depending on the OS and virtual memory settings).
When capturing a memory snapshot, JVM of version 1.3/1.4 can crash with an error like this:
Exception java.lang.OutOfMemoryError: requested 291955644 bytes for unsigned char in D:/BUILD_AREA/jdk142-update/ws/fcs/hotspot\src\share\vm\prims\jvmpi.cpp. Out of swap space?
The problem is that Java versions earlier than 5 were, by design, not capable of capturing memory snapshots for big heaps. The actual size may differ depending on the kind of objects in the heap, but in our experience, the limit is usually 1 GB.
Technically, the out of memory error happens when the JVM internally allocates a continuous block of memory and fills it with heap data, to pass it to a profiler. The size of memory needed for this temporary structure can be compared with the size of the heap.
This problem cannot be solved in 1.3/1.4 JVMs.