how to make Sliding window model for data stream mining?

Question

we have a situation that a stream (data from sensor or click stream data at server) is coming with sliding window algorithm we have to store the last (say) 500 samples of data in memory. These samples are then used to create histograms, aggregations & capture information about anomalies in the input data stream.

please tell me how to make such sliding window.

Answer 1

If you are asking how to store and maintain these values in a sliding-window manner, consider this simple example which keep tracks of the running mean of the last 10 values of some random stream of data:

WINDOW_SIZE = 10;
x = nan(WINDOW_SIZE,1);

%# init
counter = 0;
stats = [NaN NaN];                    %# previous/current value

%# prepare figure
SHOW_LIM = 200;
hAx = axes('XLim',[1 SHOW_LIM], 'YLim',[200 800]);
hLine = line('XData',1, 'YData',nan, 'EraseMode','none', 'Paren',hAx);

%# infinite loop!
while true
   val = randi([1 1000]);            %# get new value from data stream
   x = [ x(2:end) ; val ];           %# add to window in a cyclic manner
   counter = counter + 1;

   %# do something interesting with x
   stats(1) = stats(2);              %# keep track of the previous mean
   stats(2) = nanmean(x);            %# update the current mean

   %# show and update plot
   set(hLine, 'XData',[counter-1 counter], 'YData',[stats(1) stats(2)])
   if rem(counter,SHOW_LIM)==0
      %# show only the last couple of means
      set(hAx, 'XLim', [counter counter+SHOW_LIM]);
   end
   drawnow
   pause(0.02)
   if ~ishandle(hAx), break, end     %# break in case you close the figure
end