multi-core processing in R on windows XP - via doMC and foreach

Question

Hi guys,

I'm posting this question to ask for advice on how to optimize the use of multiple processors from R on a Windows XP machine.

At the moment I'm creating 4 scripts (each script with e.g. for (i in 1:100) and (i in 101:200), etc) which I run in 4 different R sessions at the same time. This seems to use all the available cpu.

I however would like to do this a bit more efficient. One solution could be to use the "doMC" and the "foreach" package but this is not possible in R on a Windows machine.

e.g.

library("foreach")
library("strucchange")
library("doMC") # would this be possible on a windows machine?
registerDoMC(2)  # for a computer with two cores (processors)
## Nile data with one breakpoint: the annual flows drop in 1898
## because the first Ashwan dam was built
data("Nile")
plot(Nile)

## F statistics indicate one breakpoint
fs.nile <- Fstats(Nile ~ 1)
plot(fs.nile)
breakpoints(fs.nile)     # , hpc = "foreach" --> It would be great to test this.
lines(breakpoints(fs.nile))

Any solutions or advice?

Thanks, Jan

Answer 1

Try the doSNOW parallel backend- it is supported out of the box on Windows. Use it with a snow socket cluster.

Answer 2

You could try doSMP from REvolution Computing. For more information, see this blog posting: Parallel Multicore Processing with R (on Windows)

Answer 3

For completeness, here is the requested answer to Tal's comment which provides a simple and portable alternative. The answer consists of running

 > library(snow)
 > help(makeCluster)

and running the first three lines of code from the top of the Examples: section:

> cl <- makeCluster(c("localhost","localhost"), type = "SOCK")
> clusterApply(cl, 1:2, get("+"), 3)
[[1]]
[1] 4

[[2]]
[1] 5

> stopCluster(cl)
> .Platform$OS.type
[1] "windows"
> 

Was that really that hard?

Add-on packages like doSNOW and thereafter foreach can make use of this in a portable way.

Answer 4

Hi guys,

Thanks a lot for all the feedback. I have tried the example above with the following code:

but without any difference in performance (even slower) as measured with system.time() on windows XP os:

library("foreach")
library("strucchange")
data("UKDriverDeaths")
seatbelt <- log10(UKDriverDeaths)
seatbelt <- cbind(seatbelt, lag(seatbelt, k = -1), lag(seatbelt, k = -12))
colnames(seatbelt) <- c("y", "ylag1", "ylag12")
seatbelt <- window(seatbelt, start = c(1970, 1), end = c(1984,12))

# without
system.time(
for (i in 1:10) {
  print(system.time(bp.seat <- breakpoints(y ~ ylag1 + ylag12, data = seatbelt, h = 0.1)))
})
## with SNOW 
library(snow)
cl <- makeCluster(c("localhost","localhost"), type = "SOCK")
clusterApply(cl, 1:2, get("+"), 3)
system.time(
for (i in 1:10) {
  print(system.time(bp.seat <- breakpoints(y ~ ylag1 + ylag12, data = seatbelt, h = 0.1,hpc=c("foreach"))))
}
)
stopCluster(cl)

Any solutions or advice? Should I measure this differently?

thanks and cheers, Jan