Automated script to zip IIS logs?

Question

I'd like to write a script/batch that will bunch up my daily IIS logs and zip them up by month.

ex080801.log which is in the format of ex*yymmdd*.log

ex080801.log - ex080831.log gets zipped up and the log files deleted.

The reason we do this is because on a heavy site a log file for one day could be 500mb to 1gb so we zip them up which compresses them by 98% and dump the real log file. We use webtrend to analyze the log files and it is capable of reading into a zip file.

Does anyone have any ideas on how to script this or would be willing to share some code?

Answer 1

Regex will do the trick... create a perl/python/php script to do the job for you..
I'm pretty sure windows batch file can't do regex.

Answer 2

You'll need a command line tool to zip up the files. I recommend 7-Zip which is free and easy to use. The self-contained command line version (7za.exe) is the most portable choice.

Here's a two-line batch file that would zip the log files and delete them afterwards:

7za.exe a -tzip ex%1-logs.zip %2\ex%1*.log
del %2\ex%1*.log

The first parameter is the 4 digit year-and-month, and the second parameter is the path to the directory containing your logs. For example: ziplogs.bat 0808 c:\logs

It's possible to get more elaborate (i.e. searching the filenames to determine which months to archive). You might want to check out the Windows FINDSTR command for searching input text with regular expressions.

Answer 3

We use a script like the following. Gzip is from the cygwin project. I'm sure you could modify the syntax to use a zip tool instead. The "skip" argument is the number of files to not archive off -- we keep 11 days in the 'current' directory.

@echo off
setlocal
For /f "skip=11 delims=/" %%a in ('Dir D:\logs\W3SVC1\*.log /B /O:-N /T:C')do move "D:\logs\W3SVC1\%%a" "D:\logs\W3SVC1\old\%%a"
d:
cd "\logs\W3SVC1\old"
gzip -n *.log
Endlocal
exit

Answer 4

Here's my script which basically adapts David's, and zips up last month's logs, moves them and deletes the original log files. this can be adapted for Apache logs too. The only problem with this is you may need to edit the replace commands, if your DOS date function outputs date of the week. You'll also need to install 7-zip.

You can also download IISlogslite but it compresses each day's file into a single zip file which I didn't find useful. There is a vbscript floating about the web that does the same thing.

-------------------------------------------------------------------------------------
@echo on

:: Name - iislogzip.bat
:: Description - Server Log File Manager
::
:: History
:: Date         Authory      Change
:: 27-Aug-2008  David Crow   Original (found on stack overflow)
:: 15-Oct-2008  AIMackenzie  Slimmed down commands


:: ========================================================
:: setup variables and parameters
:: ========================================================
:: generate date and time variables

set month=%DATE:~3,2%
set year=%DATE:~8,2%

::Get last month and check edge conditions

set /a lastmonth=%month%-1
if %lastmonth% equ 0 set /a year=%year%-1
if %lastmonth% equ 0 set lastmonth=12
if %lastmonth% lss 10 set lastmonth=0%lastmonth%

set yymm=%year%%lastmonth%

set logpath="C:\WINDOWS\system32\LogFiles"
set zippath="C:\Program Files\7-Zip\7z.exe"
set arcpath="C:\WINDOWS\system32\LogFiles\WUDF"


:: ========================================================
:: Change to log file path
:: ========================================================
cd /D %logpath%

:: ========================================================
:: zip last months IIS log files, move zipped file to archive 
:: then delete old logs
:: ========================================================
%zippath% a -tzip ex%yymm%-logs.zip %logpath%\ex%yymm%*.log
move "%logpath%\*.zip" "%arcpath%"
del %logpath%\ex%yymm%*.log

Answer 5

You can grab the command-line utilities package from DotNetZip to get tools to create zips from scripts. There's a nice little tool called Zipit.exe that runs on the command line, adds files or directories to zip files. It is fast, efficient.

A better option might be to just do the zipping from within PowerShell.

function ZipUp-Files ( $directory )
{

  $children = get-childitem -path $directory
  foreach ($o in $children) 
  {
    if ($o.Name -ne "TestResults" -and 
        $o.Name -ne "obj" -and 
        $o.Name -ne "bin" -and 
        $o.Name -ne "tfs" -and 
        $o.Name -ne "notused" -and 
        $o.Name -ne "Release")
    {
      if ($o.PSIsContainer)
      {
        ZipUp-Files ( $o.FullName )
      }
      else 
      {
        if ($o.Name -ne ".tfs-ignore" -and
           !$o.Name.EndsWith(".cache") -and
           !$o.Name.EndsWith(".zip") )
        {
          Write-output $o.FullName
          $e= $zipfile.AddFile($o.FullName)
        }
      }
    }
  }
}


[System.Reflection.Assembly]::LoadFrom("c:\\\bin\\Ionic.Zip.dll");

$zipfile =  new-object Ionic.Zip.ZipFile("zipsrc.zip");

ZipUp-Files "DotNetZip"

$zipfile.Save()

Answer 6

Borrowed zip function from http://blogs.msdn.com/daiken/archive/2007/02/12/compress-files-with-windows-powershell-then-package-a-windows-vista-sidebar-gadget.aspx

Here is powershell answer that works wonders:

param([string]$Path = $(read-host "Enter the path"))
function New-Zip
{
    param([string]$zipfilename)
    set-content $zipfilename ("PK" + [char]5 + [char]6 + ("$([char]0)" * 18))
    (dir $zipfilename).IsReadOnly = $false
}
function Add-Zip
{
    param([string]$zipfilename)

    if(-not (test-path($zipfilename)))
    {
        set-content $zipfilename ("PK" + [char]5 + [char]6 + ("$([char]0)" * 18))
        (dir $zipfilename).IsReadOnly = $false  
    }

    $shellApplication = new-object -com shell.application
    $zipPackage = $shellApplication.NameSpace($zipfilename)

    foreach($file in $input) 
    { 
            $zipPackage.CopyHere($file.FullName)
            Start-sleep -milliseconds 500
    }
}
$FilesToZip = dir $Path -recurse -include *.log
foreach ($file in $FilesToZip) {
New-Zip $file.BaseName
dir $($file.directoryname+"\"+$file.name) | Add-zip $($file.directoryname+"\$($file.basename).zip")
del $($file.directoryname+"\"+$file.name)
}