Python - calculate multinomial probability density functions on large dataset?

Question

Hi,

I originally intended to use MATLAB to tackle this problem but the in-built function has limitations that do not suit my goal. The same limitation occurs in NumPy.

I have two tab-delimited files. The first is a file showing amino acid residue, frequency and count for an in-house database of protein structures, i.e.

A    0.25    1
S    0.25    1
T    0.25    1
P    0.25    1

The second file consists of quadruplets of amino acids and the number of times they occur, i.e.

ASTP    1

Note, there are >8,000 such quadruplets.

Based on the background frequency of occurence of each amino acid and the count of quadruplets, I aim to calculate the multinomial probability density function for each quadruplet and subsequently use it as the expected value in a maximum likelihood calculation.

The multinomial distribution is as follows:

f(x|n, p) = n!/(x1!*x2!*...*xk!)*((p1^x1)*(p2^x2)*...*(pk^xk))

where x is the number of each of k outcomes in n trials with fixed probabilities p. n is 4 four in all cases in my calculation.

I have created four functions to calculate this distribution.

# functions for multinomial distribution


def expected_quadruplets(x, y):
    expected = x*y
    return expected

# calculates the probabilities of occurence raised to the number of occurrences

def prod_prob(p1, a, p2, b, p3, c, p4, d):
    prob_prod = (pow(p1, a))*(pow(p2, b))*(pow(p3, c))*(pow(p4, d))
    return prob_prod 


# factorial() and multinomial_coefficient() work in tandem to calculate C, the multinomial coefficient

def factorial(n):
    if n <= 1:
        return 1
    return n*factorial(n-1)


def multinomial_coefficient(a, b, c, d):
    n = 24.0
    multi_coeff =  (n/(factorial(a) * factorial(b) * factorial(c) * factorial(d)))
    return multi_coeff

The problem is how best to structure the data in order to tackle the calculation most efficiently, in a manner that I can read (you guys write some cryptic code :-)) and that will not create an overflow or runtime error.

To date my data is represented as nested lists.

amino_acids = [['A', '0.25', '1'], ['S', '0.25', '1'], ['T', '0.25', '1'], ['P', '0.25', '1']]

quadruplets = [['ASTP', '1']]

I initially intended calling these functions within a nested for loop but this resulted in runtime errors or overflow errors. I know that I can reset the recursion limit but I would rather do this more elegantly.

I had the following:

for i in quadruplets:
    quad = i[0].split(' ')
    for j in amino_acids:
        for k in quadruplets:
            for v in k:
                if j[0] == v:
                    multinomial_coefficient(int(j[2]), int(j[2]), int(j[2]), int(j[2]))

I haven'te really gotten to how to incorporate the other functions yet. I think that my current nested list arrangement is sub optimal.

I wish to compare each letter within the string 'ASTP' with the first component of each sub list in amino_acids. Where a match exists, I wish to pass the appropriate numeric values to the functions using indices.

Is their a better way? Can I append the appropriate numbers for each amino acid and quadruplet to a temporary data structure within a loop, pass this to the functions and clear it for the next iteration?

Thanks, S :-)

Answer 1

This might be tangential to your original question, but I strongly advise against calculating factorials explicitly due to overflows. Instead, make use of the fact that factorial(n) = gamma(n+1), use the logarithm of the gamma function and use additions instead of multiplications, subtractions instead of divisions. scipy.special contains a function named gammaln, which gives you the logarithm of the gamma function.

from itertools import izip
from numpy import array, log, exp
from scipy.special import gammaln

def log_factorial(x):
    """Returns the logarithm of x!
    Also accepts lists and NumPy arrays in place of x."""
    return gammaln(array(x)+1)

def multinomial(n, xs, ps):
    xs, ps = array(xs), array(ps)
    result = log_factorial(n) - sum(log_factorial(xs)) + sum(xs * log(ps))
    return exp(result)

If you don't want to install SciPy just for the sake of gammaln, here is a replacement in pure Python (of course it is slower and it is not vectorized like the one in SciPy):

def gammaln(n):
    """Logarithm of Euler's gamma function for discrete values."""
    if n < 1:
        return float('inf')
    if n < 3:
        return 0.0
    c = [76.18009172947146, -86.50532032941677, \
         24.01409824083091, -1.231739572450155, \
         0.001208650973866179, -0.5395239384953 * 0.00001]
    x, y = float(n), float(n)
    tm = x + 5.5
    tm -= (x + 0.5) * log(tm)
    se = 1.0000000000000190015
    for j in range(6):
        y += 1.0
        se += c[j] / y
    return -tm + log(2.5066282746310005 * se / x)

Another easy trick is to use a dict for amino_acids, indexed by the residue itself. Given your original amino_acids structure, you can do this:

amino_acid_dict = dict((amino_acid[0], amino_acid) for amino_acid in amino_acids)
print amino_acid_dict
{"A": ["A", 0.25, 1], "S": ["S", 0.25, 1], "T": ["T", 0.25, 1], "P": ["P", 0.25, 1]}

You can then look up the frequencies or counts by residue easier:

freq_A = amino_acid_dict["A"][1]
count_A = amino_acid_dict["A"][2]

This saves you some time in the main loop:

for quadruplet in quadruplets:
    probs = [amino_acid_dict[aa][1] for aa in quadruplet]
    counts = [amino_acid_dict[aa][2] for aa in quadruplet]
    print quadruplet, multinomial(n, probs, counts)