merging and manupulating files in matlab

Question

Is there a way to run a loop through a folder and process like 30 files for a month and give the average,max of each columns and write in one excel sheet or so??

I have 30 files of size [43200 x 30]
I ran a different matlab scrip to generate them so the names are easy File_2010_04_01.xls , File_2010_04_02.xls ..... and so on I cannot merge them as each are 20mbs and matlab would crash. Any ideas? Thanks

Answer 1

You can use the dir command to get a list (it is actually a structure array) of the files in a given folder. That should allow you to access your files and perform whatever operation you need to do. Additionally, since you generated the filename yourself, there should be no problem in re-generating them.

If you're worried about the size of the data you might have to read, consider using the optional argument of fscanf that can limit the size of the read data. You can also use fgetl to read and process your files line by line.

Finally, for such operations on text files, there might be better tools than Matlab out there.

Answer 2

You can first get a list of your files using the function DIR. Here's an example:

dirData = dir('File_2010_04_*.xls');  %# Match file names with a wildcard
dataFiles = {dirData.name};           %# Get the file names in a cell array

Once you have these files, you can loop over them using XLSREAD to load the data. Note XLSREAD can return different versions of the data in the Excel file:

[numData,txtData,rawData] = xlsread(fileName);  %# Where fileName is a string

Here, numData contains the cells with numeric data in the file, txtData contains the cells with text data in the file, and rawData is a cell array that contains all of the data in the file. You will have to determine which data array to use and how to index it to get your 43200-by-30 matrix of data to process.

Putting this together, here's a code sample for how you could process your data to get the column maxima and column averages across all of your files:

columnTotal = zeroes(1,30);             %# Initialize column sum
columnMax = -inf(1,30);                 %# Initialize column maxima
dirData = dir('File_2010_04_*.xls');    %# Match file names in the current folder
dataFiles = {dirData.name};             %# Get the file names in a cell array
nFiles = numel(dataFiles);              %# Number of files
for iFile = 1:nFiles                    %# Loop over the files
  numData = xlsread(dataFiles{iFile});  %# Load the data
  %# Here, I'm assuming "numData" contains your 43200-by-30 matrix
  columnTotal = columnTotal+sum(numData);     %# Add up column data
  columnMax = max(columnMax,max(numData));    %# Get the column maxima
end
columnAverage = columnTotal./(nFiles*43200);  %# Average across all files