Limit loop to N iterations per minute

Question

Hi,

Could someone please give me some help with limiting a loop to N iterations per minute in Python.

Lets say I have

limit = 5
for items in recvData:
     # > limit iterations in past minute? -> sleep for 60 seconds from last iterations before proceeding? #
     ... do work ...

How would I do the time check / sleeping to give the correct flow. I'm not worried about blocking the executing thread/process while it waits.

Thanks

Answer 1

Use grouper from the itertools recipes, combined with a time check.

import itertools, datetime, time
limit = 5

def grouper(n, iterable, fillvalue=None):
    "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return itertools.izip_longest(fillvalue=fillvalue, *args)

for items in grouper(limit, recvData):
    prev_minute = datetime.datetime.now().minute

    for item in items:
        # do stuff

    _, _, _, _, minute, second, _ = datetime.datetime.now()
    if minute == prev_minute:
        time.sleep( 60 - second )

Answer 2

It should be noted that this is not "hard real time" code. this will be off slightly because of OS scheduling and the like. That being said, unless you know that you need hard real time, this should suffice.

import time

limit = 5
starttime = time.time()
for i, item in enumerate(recvData):
    if not i + 1 % limit:
        sleeptime =starttime + 60 - time.time()
        if sleeptime > 0:
            time.sleep(sleeptime)
        starttime = time.time()
    #processing code

Answer 3

This very much depends on the type of work you're doing inside the loop and on how accurate you want this mechanism to work.

ATM I can come up with 2 possible schemes:

• If one iteration takes about constant time than you could calculate an average from the first few iterations and "sleep" for 1 - iterationTime afterwards.
• Otherwise, you can poll the time and recalculate the average every step (or a few steps).

Depending on the standard deviation of your single loop execution times both scheme can work quite well, but if the execution times are very varying, neither of them will. Also, if you want evenly distributed loop cycles and not only keep the average/min you have to distribute the sleep-s and do one after each iteration.

I am not familiar enough with Python to know how expensive is to query the time and what other Python-specific issues might pop up with sleep-ing, though.