Hello,
Transferring the memory snapshots between the production systems / my own machine can take hours because of the network configurations (intermediary bastions, moving data from us-west-1 to eastern EU) coupled with large heaps. Are there any plans to use a better compression algorithms/reduce compressed snapshot size even more ? Eg applyin zstd (3 level) shrinks the already compressed snapshot from 2.16gb to 0.85gb
Thanks,
Cornel.
Improve snapshot compression
-
- Posts: 1626
- Joined: Tue Aug 10, 2004 7:52 pm
Re: Improve snapshot compression
Profiler offers to transfer compressed snapshot. Did you try this option? We are using deflate (compression level 1) which offers great compromise between speed and compression level (about 3).
Best regards,
Vladimir Kondratyev
YourKit GmbH
http://www.yourkit.com
"Don't get lost in data, get information!"
Best regards,
Vladimir Kondratyev
YourKit GmbH
http://www.yourkit.com
"Don't get lost in data, get information!"
-
- Posts: 4
- Joined: Tue Apr 04, 2023 3:10 pm
Re: Improve snapshot compression
Hi Vladimir,
Yes I used that, the number that I showed you (2.16) was obtained using a compressed snapshot.
Deflate was the king up until 6-7 years ago, now most of the people handling large data are using zstd (level 3) or lz4.
Cornel.
Yes I used that, the number that I showed you (2.16) was obtained using a compressed snapshot.
Deflate was the king up until 6-7 years ago, now most of the people handling large data are using zstd (level 3) or lz4.
Cornel.
-
- Posts: 1626
- Joined: Tue Aug 10, 2004 7:52 pm
Re: Improve snapshot compression
What is the size of original snapshot? It look unbeivable that zstd give 3x better compression that deflate, and I doubt that zstd can compress already compressed data.