Apart from graphical estimation of linearity (gaze-at-scatterplot method), which is utilized before applying some technique from GLM family, there are several ways to do this estimation arithmetically (i.e. without graphs).
Right now, I'll focus on Fisher's eta-squared - correlation ratio: arithmetically, it's equal to squared Pearson's r (coef. of determination: r2) if relationship between two variables is linear. Hence, you can compare values of eta and r and make an assessment about type of relation (linear or not). It provides an information about percent of variance in the dependent variable explained (linearly or not) by the independent variable. Therefore, you can apply it when linearity assumptions are not met.
Simply stated: is there a routine for eta/eta-squared in R?