Improve snapshot compression

Questions about YourKit Java Profiler
Locked
cornelc
Posts: 4
Joined: Tue Apr 04, 2023 3:10 pm

Improve snapshot compression

Post by cornelc »

Hello,

Transferring the memory snapshots between the production systems / my own machine can take hours because of the network configurations (intermediary bastions, moving data from us-west-1 to eastern EU) coupled with large heaps. Are there any plans to use a better compression algorithms/reduce compressed snapshot size even more ? Eg applyin zstd (3 level) shrinks the already compressed snapshot from 2.16gb to 0.85gb

Thanks,
Cornel.
Vladimir Kondratyev
Posts: 1624
Joined: Tue Aug 10, 2004 7:52 pm

Re: Improve snapshot compression

Post by Vladimir Kondratyev »

Profiler offers to transfer compressed snapshot. Did you try this option? We are using deflate (compression level 1) which offers great compromise between speed and compression level (about 3).

Best regards,
Vladimir Kondratyev
YourKit GmbH
http://www.yourkit.com
"Don't get lost in data, get information!"
cornelc
Posts: 4
Joined: Tue Apr 04, 2023 3:10 pm

Re: Improve snapshot compression

Post by cornelc »

Hi Vladimir,

Yes I used that, the number that I showed you (2.16) was obtained using a compressed snapshot.

Deflate was the king up until 6-7 years ago, now most of the people handling large data are using zstd (level 3) or lz4.

Cornel.
Vladimir Kondratyev
Posts: 1624
Joined: Tue Aug 10, 2004 7:52 pm

Re: Improve snapshot compression

Post by Vladimir Kondratyev »

What is the size of original snapshot? It look unbeivable that zstd give 3x better compression that deflate, and I doubt that zstd can compress already compressed data.
Locked