Iterating over the big matrix containing 3000 rows and calculate the correlation.-Follow-up!

Question

Thanks Nico! Almost got there after I corrected small bugs. Here I attache my script:

datamatrix=read.table("ref.txt", sep="\t", header=T, row.names=1)
correl <- NULL
for (i in 1:nrow(datamatrix)) {
   correl <- apply(datamatrix, 1, function(x) {cor(t(datamatrix[, i]))})
   write.table(correl, paste(row.names(datamatrix)[i], ".txt", sep=""))
}

But I am afraid the function(x) part is of problem, that seems to be t(datamatrix[i,j]), which will calculate corr of any two rows.

Actually I need to iterate through the matrix. first cor(row01, row02) get one correlation between rwo01 and row02; then cor(row01, row03) to get the correlation of row01 and rwo03, ....and till correlation between row01 row30000. Now I got the first column for row01 Row01 1.000 Row02 0.012 Row03 0.023 Row04 0.820 Row05 0.165 Row06 0.230 Row07 0.376 Row08 0.870 and save it to file row01.txt.

Similarly get Row02 Row01 0.012 Row02 1.000 Row03 0.023 Row04 0.820 Row05 0.165 Row06 0.230 Row07 0.376 Row08 0.870 and save it to file row02.txt.

Totally I will get 30000 files. It is stupid, but this can skip the memory limit and can be easily handled for the correlation of a specific row.

Answer 1

First, your code is erroneous : The correl step should be :

correl <- apply(datamatrix, 1, function(x) {cor(x,datamatrix[i, ])})

Then, you better close the connections explicitly, otherwise R may leave too many connections open.

Finally, using write.table doesn't guarantee that you can retrieve the data easily. You have to construct a table yourself. Try this code :

correl <- NULL
XX <- for (i in 1:nrow(datamatrix)) {
   correl <- apply(datamatrix, 1, function(x) {cor(x,datamatrix[i, ])})
   ff <- file(paste(row.names(datamatrix)[i], ".txt", sep=""),open="at")
      write(paste("row","cor"),ff)
      tmp <- paste(names(correl),correl)
      write(tmp, ff,sep="\n")
   }
   close(ff)
}