Yes, you can do this with JavaScript. No, you don't need jQuery.
Here's one way:
function howManyAreChecked()
{
one = document.getElementById("one").checked;
two = document.getElementById("two").checked;
three= document.getElementById("three").checked;
var checkCount = 0;
if ( one == true )
{ checkCount = checkCount + 1 }
if ( two == true )
{ checkCount = checkCount + 1 }
if ( three == true )
{ checkCount = checkCount + 1 }
alert(checkCount);
}
The example above assumes that you've got 3 checkboxes in HTML, with ids "one", "two" and "three". The script first stores the value of the "checked" property, which can be either TRUE or FALSE. The script then looks at each different variable, and if TRUE, increments the counter.
There could many causes why your code is giving you an undefined error. If you can post your code - either a link to your page, or all or HTML and all your JavaScript - then we can take a look at that.