converting uneven hierarchical list to a data frame

Question

I don't think this has been asked yet, but is there a way to combine information of a list with multiple levels and uneven structure into a data frame of "long" format?

Specifically:

library(XML)
library(plyr)
xml.inning <- "http://gd2.mlb.com/components/game/mlb/year_2009/month_05/day_02/gid_2009_05_02_chamlb_texmlb_1/inning/inning_5.xml"
xml.parse <- xmlInternalTreeParse(xml.inning)
xml.list <- xmlToList(xml.parse)
## $top$atbat
## $top$atbat$pitch
##             des              id            type               x               y 
##          "Ball"           "310"             "B"         "70.39"        "125.20" 

Where the following is the structure:

> llply(xml.list, function(x) llply(x, function(x) table(names(x))))
$top
$top$atbat
.attrs  pitch 
     1      4 
$top$atbat
.attrs  pitch 
     1      4 
$top$atbat
.attrs  pitch 
     1      5 
$bottom
$bottom$action
     b    des  event      o  pitch player      s 
     1      1      1      1      1      1      1 
$bottom$atbat
.attrs  pitch 
     1      5 
$bottom$atbat
.attrs  pitch 
     1      5 
$bottom$atbat
.attrs  pitch runner 
     1      5      1 
$bottom$atbat
.attrs  pitch runner 
     1      7      1 
$.attrs
$.attrs$num
character(0)
$.attrs$away_team
character(0)
$.attrs$

What I'd like to have is a data frame from the named vector from the pitch category, along with the proper (top, atbat, bottom). Therefore, I would need to ignore levels that won't fit into a data.frame due to different number of columns. Something like this:

   first second third    des     x
1    top  atbat pitch   Ball 70.29
2    top  atbat pitch Strike 69.24
3 bottom  atbat pitch    Out 67.22

Is there an elegant way of doing this? Thanks!

Answer 1

I don't know about elegant, but this works. Those more familiar with plyr could probably provide a more general solution.

> cleanFun <- function(x) {
+   a <- x[["atbat"]]
+   b <- do.call(rbind,a[names(a)=="pitch"])
+   c <- as.data.frame(b)
+ }
> ldply(xml.list[c("top","bottom")], cleanFun)[,1:5]
     .id             des  id type      x
1    top            Ball 310    B  70.39
2    top   Called Strike 311    S 118.45
3    top   Called Strike 312    S  86.70
4    top In play, out(s) 313    X  79.83
5 bottom            Ball 335    B  15.45
6 bottom   Called Strike 336    S  77.25
7 bottom Swinging Strike 337    S  99.57
8 bottom            Ball 338    B 106.44
9 bottom In play, out(s) 339    X 134.76

Answer 2

The .id feature for the ldply() is nice, but it seems like they overlap once you do another ldply().

Here is fairly general function that uses rbind.fill():

aho <- ldply(llply(xml.list[[1]], function(x) ldply(x, function(x) rbind.fill(data.frame(t(x))))))
> aho[1:5,1:4]
     .id                                                       des   id type
1  pitch                                                      Ball  310    B
2  pitch                                             Called Strike  311    S
3  pitch                                             Called Strike  312    S
4  pitch                                           In play, out(s)  313    X
5 .attrs Alexei Ramirez lines out to second baseman Ian Kinsler.   <NA> <NA>

The .id for the second ldply() is missing because we already had an .id. We could fix this by naming the first .id as a different name, but it doesn't seem coherent.

aho2 <- ldply(llply(xml.list[[1]], function(x) {
  out <- ldply(x, function(x) rbind.fill(data.frame(t(x))))
  names(out)[1] <- ".id2"
  out
}))
> aho2[1:5,1:4]
    .id   .id2                                                       des   id
1 atbat  pitch                                                      Ball  310
2 atbat  pitch                                             Called Strike  311
3 atbat  pitch                                             Called Strike  312
4 atbat  pitch                                           In play, out(s)  313
5 atbat .attrs Alexei Ramirez lines out to second baseman Ian Kinsler.   <NA>