How to limit I/O operations in .NET application?

Question

I'm developing an application (.NET 4.0, C#) that:
1. Scans file system.
2. Opens and reads some files.

The app will work in background and should have low impact on the disk usage. It shouldn't bother users if they are doing their usual tasks and the disk usage is high. And vice versa, the app can go faster if nobody is using the disk.
The main issue is I don't know real amount and size of I/O operations because of using API (mapi32.dll) to read files. If I ask API to do something I don't know how many bytes it reads to handle my response.

So the question is how to monitor and manage the disk usage? Including file system scanning and files reading...
Check performance counters that are used by standard Performance Monitor tool? Or any other ways?

Thanks

Answer 1

Check if the screensaver is running ? Good indication that the user is away from the keyboard

Answer 2

See this question and this also for related queries. I would suggest for a simple solution just querying for the current disk & CPU usage % every so often, and only continue with the current task when they are under a defined threshold. Just make sure your work is easily broken into tasks, and that each task can be easily & efficiently start/stopped.

Answer 3

Use the Sytem.Diagnostics.PerformanceCounter class and attach to the PhysicalDisk counter for the drive that you are indexing.

Below is some code from disklight.codeplex.com, although its currently hard-wired to the "C:" drive. You will want to change "C:" to whichever drive your process is scanning.

The counter's I would watch would probably be "% Idle Time", which pretty much says how often the drive is doing anything. 0% idle just means the disk is busy, but does not necessarily mean that it is flat-out and cannot transfer more data.

Combine the % Idle Time with "Current Disk Queue Length" and this will tell you if the drive is getting so busy that it cannot service all the requests for data. As guidelines, anything over 0 means drive is probably flat-out busy, and anything over 2 means the drive is completely saturated. These would probably apply to both SSD and HDD fairly well.

Also, any value that you read is an instantaneous value at a point in time. You should do a running average over a few results, e.g. take a reading every 100ms, and average 5 readings before using the information from the result to make a decision... like waiting until the counters settle before making your next IO request.

public struct DiskUsageStats
{
    public int DiskQueueLength;
    public int DiskUsagePercent;
    public string DriveName;
    public int ReadBytesPerSec;
    public int WriteBytesPerSec;
}

internal class DiskUsageMonitor
{
    private readonly PerformanceCounter _diskQueueCounter;
    private readonly PerformanceCounter _idleCounter;
    private Timer _perfTimer;
    private readonly PerformanceCounter _readBytesCounter;
    private int _updateResolutionMillisecs = 100;
    private readonly PerformanceCounter _writeBytesCounter;

    /// <summary>
    /// Initializes a new instance of the <see cref="DiskUsageMonitor"/> class.
    /// </summary>
    /// <param name="updateResolutionMillisecs">The update resolution millisecs.</param>
    internal DiskUsageMonitor(int updateResolutionMillisecs)
        : this(null)
    {
        _updateResolutionMillisecs = updateResolutionMillisecs;
    }

    /// <summary>
    /// Initializes a new instance of the <see cref="DiskUsageMonitor"/> class.
    /// </summary>
    /// <param name="updateResolutionMillisecs">The update resolution millisecs.</param>
    /// <param name="driveName">Name of the drive.</param>
    internal DiskUsageMonitor(int updateResolutionMillisecs, string driveName)
        : this(driveName)
    {
        _updateResolutionMillisecs = updateResolutionMillisecs;
    }

    /// <summary>
    /// Initializes a new instance of the <see cref="DiskUsageMonitor"/> class.
    /// </summary>
    /// <param name="driveName">Name of the drive.</param>
    internal DiskUsageMonitor(string driveName)
    {

        // Get a list of the counters and look for "C:"

        var perfCategory = new PerformanceCounterCategory("PhysicalDisk");
        string[] instanceNames = perfCategory.GetInstanceNames();
        foreach (string name in instanceNames)
        {
            if (name.IndexOf("C:") > 0)
            {
                if (string.IsNullOrEmpty(driveName))
                    driveName = name;
            }
        }


        _readBytesCounter = new PerformanceCounter("PhysicalDisk", "Disk Read Bytes/sec", driveName);
        _writeBytesCounter = new PerformanceCounter("PhysicalDisk", "Disk Write Bytes/sec", driveName);
        _diskQueueCounter = new PerformanceCounter("PhysicalDisk", "Current Disk Queue Length", driveName);
        _idleCounter = new PerformanceCounter("PhysicalDisk", "% Idle Time", driveName);
        InitTimer();
    }

    /// <summary>
    /// Default constructor
    /// </summary>
    internal DiskUsageMonitor()
        : this(null)
    {
    }

    /// <summary>
    /// Gets or sets the update interval milli secs.
    /// </summary>
    /// <value>The update interval milli secs.</value>
    public int UpdateIntervalMilliSecs
    {
        get { return _updateResolutionMillisecs; }
        set
        {
            _updateResolutionMillisecs = value;
            InitTimer();
        }
    }

    internal event DiskUsageResultHander DiskUsageResult;

    /// <summary>
    /// Inits the timer.
    /// </summary>
    private void InitTimer()
    {
        StopTimer();
        _perfTimer = new Timer(_updateResolutionMillisecs);
        _perfTimer.Elapsed += PerfTimerElapsed;
        _perfTimer.Start();
    }

    /// <summary>
    /// Performance counter timer elapsed event handler
    /// </summary>
    /// <param name="sender">The sender.</param>
    /// <param name="e">The <see cref="System.Timers.ElapsedEventArgs"/> instance containing the event data.</param>
    private void PerfTimerElapsed(object sender, ElapsedEventArgs e)
    {
        float diskReads = _readBytesCounter.NextValue();
        float diskWrites = _writeBytesCounter.NextValue();
        float diskQueue = _diskQueueCounter.NextValue();
        float idlePercent = _idleCounter.NextValue();

        if (idlePercent > 100)
        {
            idlePercent = 100;
        }

        if (DiskUsageResult != null)
        {
            var stats = new DiskUsageStats
                            {
                                DriveName = _readBytesCounter.InstanceName,
                                DiskQueueLength = (int)diskQueue,
                                ReadBytesPerSec = (int)diskReads,
                                WriteBytesPerSec = (int)diskWrites,
                                DiskUsagePercent = 100 - (int)idlePercent
                            };
            DiskUsageResult(stats);
        }
    }

    /// <summary>
    /// Stops the timer.
    /// </summary>
    internal void StopTimer() {
        if (_perfTimer != null) {
            try {
                _perfTimer.Stop();
            } catch {
            } finally {
                _perfTimer.Close();
                _perfTimer.Dispose();
                _perfTimer.Elapsed -= PerfTimerElapsed;
                _perfTimer = null;
            }
        }
    }

    /// <summary>
    /// Closes this instance.
    /// </summary>
    internal void Close()
    {
        StopTimer();
        DiskUsageResult = null;
    }

    #region Nested type: DiskUsageResultHander

    internal delegate void DiskUsageResultHander(DiskUsageStats diskStats);

    #endregion
}

Answer 4

A long term ago Microsoft Research published a paper on this (sorry I can’t remember the url).
From what I recall:

• The program started off doing very few "work items".
• They measured how long it took for each of their "work item".
• After running for a bit, they could work out how fast an "work item" was with no load on the system.
• From then on, if the "work item" were fast (e.g. no other programmers making requests), they made more requests, otherwise they backed-off

The basic ideal is:

“if they are slowing me down, then I must be slowing them down, so do less work if I am being slowed down”

Answer 5

Something to ponder: what if there are other processes which follow the same (or a similar) strategy? Which one would run during the "idle time"? Would the other processes get a chance to make use of the idle time at all?

Obviously this can't be done correctly unless there is some well-known OS mechanism for fairly dividing resources during idle time. In windows, this is done by calling SetPriorityClass.

This document about I/O prioritization in Vista seems to imply that IDLE_PRIORITY_CLASS will not really lower the priority of I/O requests (though it will reduce the scheduling priority for the process). Vista added new PROCESS_MODE_BACKGROUND_BEGIN and PROCESS_MODE_BACKGROUND_END values for that.

In C#, you can normally set the process priority with the Process.PriorityClass property. The new values for Vista are not available though, so you'll have to call the Windows API function directly. You can do that like this:

[DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
public static extern bool SetPriorityClass(IntPtr handle, uint priorityClass);

const uint PROCESS_MODE_BACKGROUND_BEGIN = 0x00100000;

static void SetBackgroundMode()
{
   int processId = Process.GetCurrentProcess().Id;
   SetPriorityClass(processId, PROCESS_MODE_BACKGROUND_BEGIN);
}

I did not test the code above. Don't forget that it can only work on Vista or better. You'll have to use Environment.OSVersion to check for earlier operating systems and implement a fall-back strategy.