I'm serializing a large 3d array to disk.The original data is around 50MB and GZiped output is in Kb's size.But the operation takes around 5 sec's.I would like to optimize it for time.I was thinking weather it would be any better to use a mapped read/write since i have seen it has better performance than the usual stream writing.But don't know how to use ObjectOutputStream and GZIPOutputStream along with mappped write.Please suggest weather is it worth using mapped read/write along with object output stream and also please post any sample code if any one has experience doing the same.
+1
A:
The page you linked to shows worse performance for mapped write. It only beats a regular stream on read and seek operations. So I guess this is not going to help you, as both ObjectOutputStream and GZipOutputStream are simple write-appenders.
Thilo
2010-09-08 13:04:05
Sorry not the mapped write mode.What i meant is read/write mode
Emil
2010-09-08 13:14:59
do you use read/write mode? You said you are writing something to disk using ObjectOutputStream. Sounds write-only.
Thilo
2010-09-08 13:16:58
Actually i have to read it back too after writing.This is actually 3 3D arrays of filtered data.Filtering is an expensive process and also if i keep these arrays in memory it will clog the memory so just for the time being until there is use for the filtered data it will be on disk.When required it will be read.
Emil
2010-09-08 13:21:58
Maybe you should consider another data structure. There are libraries for large arrays. Using ObjectOutputStream, you always have to read the whole thing into memory at once. Memory-mapped files will only be faster as well if you can keep most of the file in memory as well.
Thilo
2010-09-09 01:08:52
Can you mention any library that is suitable for large arrays.
Emil
2010-09-09 07:02:51
@Emil: I cannot. But someone else here sure can, ask this as a new question.
Thilo
2010-09-09 08:44:10