Python: defining new functions on the fly using "with"

Question

I want to convert the following code:

...
urls = [many urls]
links = []
funcs = []
for url in urls:
   func = getFunc(url, links)
   funcs.append(func)
...

def getFunc(url, links):
   def func():
      page = open(url)
      link = searchForLink(page)
      links.append(link)
   return func

into the much more convenient code:

urls = [many urls]
links = []
funcs = []
for url in urls:
   <STATEMENT>(funcs):
        page = open(url)
        link = searchForLink(page)
        links.append(link)

I was hoping to do this with the with statement. As I commented bellow, I was hoping to achieve:

def __enter__():
    def func():

..code in the for loop..

def __exit__():
  funcs.append(func)

Of course this doesn't work.

List comprehensions is not good for cases were the action searchForLink is not just one function but many functions. It would turn into an extremely unreadable code. For example even this would be problematic with list comprehensions:

for url in urls:
  page = open(url)
  link1 = searchForLink(page)
  link2 = searchForLink(page)
  actionOnLink(link1)
  actionOnLink(link2)
  .... many more of these actions...
  links.append(link1)

Answer 1

Lose the line <STATEMENT>(funcs):

Edit:

I mean: why would you do this? Why define a new function for each page? Why not just do this?

urls = [many urls]
links = []
for url in urls:
    page = open(url)
    link = searchForLink(page)
    links.append(link) 

Answer 2

It makes no sense to use with here. Instead use a list comprehension:

funcs = [getFunc(url, links) for url in urls]

Answer 3

There are only two ways to create functions: def and lambda. Lambdas are meant for tiny functions, so they may not be very appropriate for your case. However, if you really want to, you can enclose two lambdas within each other:

urls = [many urls]
links = []
funcs = [(lambda x:
            lambda:
              links.append(searchForLink(open(x))))(u)
         for u in urls]

A little too LISPish for my taste.

Answer 4

You should not use "with" to do this (even though, given that it's Python, you almost certainly could, using some bizarre side-effect and Python's dynamicism).

The purpose of "with" in Python is, as described in the docs, "to wrap the execution of a block with methods defined by a context manager. This allows common try...except...finally usage patterns to be encapsulated for convenient reuse."

I think you're confusing Python's "with" with the Javascript/VisualBasic "with", which may be cosmetically similar but which is effectively unrelated.

Answer 5

Good old itertools.

from itertools import imap
links.extend(imap(searchForLink, imap(open, urls)))

Although, maybe you'd prefer functional.

from functional import *
funcs = [partial(compose(compose(links.append, searchForLink), open), url) for url in urls]
for func in funcs: func()

I don't think it's worthwhile creating a class for with use: it's more work to create __enter__ and __exit__ than it is to just write a helper function.

Answer 6

A bit unconventional, but you can have a decorator register the func and bind any loop variables as default arguments:

urls = [many urls]
links = []
funcs = []

for url in urls:
    @funcs.append
    def func(url=url):
        page = open(url)
        link = searchForLink(page)
        links.append(link)

Answer 7

You may be better using generators to achieve the delayed computation you're after.

def MakeLinks(urls):
    for url in urls:
        page = open(url)
        link = searchForLink(page)
        yield link

links = MakeLinks(urls)

When you want the links:

for link in links:
    print link

The urls will be looked up during this loop, and not all at once (which it looks like you're tring to avoid).