Reading large Binary files fails in Rebol

Question

The following Rebol code fails due to an out of memory error:

read/binary http://mirror.bytemark.co.uk/ubuntu-releases/lucid/
            ubuntu-10.04-desktop-i386.iso 

How can I use Rebol to read large binary files over HTTP?

Answer 1

A straight read of a large file may fail as it is held in memory and may exhaust the available memory.

This reference describes a way to read a file in chunks, provided you can find an FTP server to source it: http://www.rebol.com/docs/core23/rebolcore-13.html#section-11.12

A quick search suggests there are FTP servers for Ubuntu, but I have not checked if they cover the version you need. Good luck!

Answer 2

Rebol 2 ports are something of a mess. So you can't directly apply the sample of how to read a large file in parts because read doesn't work on port! in R2 (much less does read/part work).

But if you jump through some hoops and use the /direct refinement to directly open the file, you get something that will work:

; Rebol 2 chunking http download sample

filename: %ubuntu-10.04-desktop-i386.iso
base-url: http://mirror.bytemark.co.uk/ubuntu-releases/lucid/
chunk-size: 32000

in-port: open/binary/direct rejoin [base-url filename]
out-port: open/binary/direct filename
while [not none? data: copy/part in-port chunk-size] [
    append out-port data
]
close in-port
close out-port

(UPDATE: I didn't notice the READ-IO sample that Sunanda cites, as I don't use R2 much and hadn't ever seen READ-IO. It may also work on http and have better performance. But I'll let you do that comparison. :P)

In Rebol 3, you can read and write from a port!. That means a modification of the large file sample should work, in theory...

; Rebol 3 chunking http download sample
; Warning: does not work in A99 release

filename: %ubuntu-10.04-desktop-i386.iso
base-url: http://mirror.bytemark.co.uk/ubuntu-releases/lucid/
chunk-size: 32000

in-port: open/read rejoin [base-url filename]
out-port: open/write filename
while [not empty? data: read/part in-port chunk-size] [
    write out-port data
]
close in-port
close out-port

Yet there's a bug in the current build of R3 alpha that ignores the /part refinement and goes ahead and returns the contents of the whole file during read if the scheme is HTTP. :(

Answer 3

Use this http://anton.wildit.net.au/rebol/util/batch-download.r to download large files.