Creating/making directories in python (complex)

Question

I am trying to create a bunch of directories/subdirectories that I can copy files into. I am working with Python and I can't seem to find a good way to do this. I have a main path that I will branch off of. Then after that, I have Weights and No_Weights. Male and Female following. Within each of Male and Female folders, I have each ethnicity (Caucasian, African-American, Asian, Hispanic, Indo, Other, Unknown). Within each of those folders, I have age ranges from Below 20, all the way to 70+ (B20, 20, 30, 40, 50, 60, 70).

I have tried to generate all of the paths so all I would have to call is mkdir about 50 times, but that is about 150 lines of code (almost).

Is there any simple way to create all of these folders without having to do it by hand?

Answer 1

Have a few nested for-loops, then os.mkdir for each one. Use os.path.join to concatenate the directory path together.

Something like:

loop weights
    mkdir weight
    loop sexes
        mkdir weights + sex
        loop ethnicities
            mkdir weights + sex + ethnicity
            loop ages
                mkdir weights + sex + ethnicity + age

here loop is just a normal for-loop:

for weight in ('weights', 'no_weights'):

mkdir is os.mkdir

'+' is os.path.join:

os.mkdir(os.path.join(weights, sex, ethnicity, age))

Edit: dir_util might be of some use here so you don't have to make each subdirectory:

http://docs.python.org/release/2.5.2/dist/module-distutils.dirutil.html

loop weights
    loop sexes
        loop ethnicities
            loop ages
                mkpath weights + sex + ethnicity + age

Answer 2

import itertools
import os

dirs = [["Weights", "No_Weights"],
        ["Male", "Female"],
        ["Caucasian", "African-American", "Asian", "Hispanic", "Indo", "Other", "Unknown"], 
        ["B20", "20", "30", "40", "50", "60", "70"]]

for item in itertools.product(*dirs):
    os.makedirs(os.path.join(*item))

itertools.product() will construct all possible path variations, then os.path.join() will join the subpaths together using the correct syntax for your platform.

EDIT: os.makedirs() is needed instead of os.mkdir(). Only the former will construct all the intermediate subdirectories in a full path.

Answer 3

This example should get you started:

import itertools
import os.path

ROOT = r'C:\base\path'

sex = ('male', 'female')
ethnicity = ('Caucasian', 'African-American', 'Asian')
ages = ('B20', '20', '30')

for path in itertools.product(sex, ethnicity, ages):
    print os.path.join(ROOT, *path)

The itertools module is your friend: http://docs.python.org/library/itertools.html#itertools.product

Answer 4

Just do something like this:

main = 'somedir'
weight = ['weights', 'No_weights']
ethnicity = ['Caucasian', #the rest]
ages = ['B20'] +  range(20, 71, 10)

for w in weights:
    os.mkdir(os.path.join(main, w)
    for e in ethnicity:
        os.mkdir(os.path.join(main, w, e))
        for a in ages:
            os.mkdir(os.path.join(main, w, e, a))

and that should take care of it for you...

Answer 5

os.makedirs can help -- it makes all intermediate directories all the way down to the "leaf" one you specify.

The other issue (generating all the "one from column A, one from column B, ..." combinations) is best approached as a problem of "counting in mixed bases" -- roughly, s/thing like...:

choices = [ ['Weights', 'Noweights'],
            ['Male', 'Female'],
            ['Caucasian', 'AfricanAmerican', ...
            ...
          ]
Ls = [len(x) for x in choices]
ct = [0] * len(Ls)

while True:
    p = [c[i] for i, c in zip(ct, choices)]
    os.makedirs(os.path.join(p))
    n = -1
    while n > -len(Ls):
      ct[n] += 1
      if ct[n] < Ls[n]: break
      ct[n] = 0
      n -= 1
    else:
      break

itertools.product is the modern and concise approach to generating all the "one from column A, etc etc" picks, and what I would advise in production software -- just:

for p in itertools.product(*choices):
    os.makedirs(os.path.join(p))

can replace all of the above code (!). I think it's also worth being aware of the "counting in mixed bases" lower-abstraction-level approach because it comes in handy in many cases (including times in which one is stuck using a Python release < 2.6;-).