There are two cases here, based on how much change you are able/allowed to introduce into the existing schema.
If you can change the schema, the simplest solution here would be to only have one field and use the scoreType field in the user record to determine how to interpret the data. If your percentage is always two decimal places then you could create your table like this:
CREATE TABLE userScore (
id INT,
userID INT,
scoreValue INT
);
And query it like this:
SELECT id,
CASE WHEN scoreType = 'points' THEN scoreValue ELSE NULL END as points,
CASE WHEN scoreType = 'percentage' THEN scoreValue / 100.0 ELSE NULL END as percentage
WHERE userID = ?
Now there's no need for a constraint of any sort.
If the user's scoreType field can change and the field (points or percentage) used would dependent on scoreType's value at the time the score is stored, you have to add scoreType to my version of userScore in order to keep the rules for interpreting that data with each record since you can't rely on the parent record being consistent (but it does eliminate the need for a join).
If you have no option to change the schema, you could do it with ON INSERT/UPDATE triggers to fetch the scoreType field to verify that the resulting record will only have a value in the proper field. In PostgreSQL (the DB I'm most familiar with), the syntax for the meat of the trigger function would be something like this (for clarity, v_* fields are variables/parameters):
-- for trigger ON INSERT/UPDATE to userScore
SELECT INTO v_scoreType scoreType FROM user WHERE userID = v_userID;
IF (v_scoreType = 'points' AND NEW.points IS NOT NULL AND NEW.percentage IS NULL)
OR (v_scoreType = 'percentage' AND NEW.points IS NULL AND NEW.percentage IS NOT NULL) THEN
RETURN NEW; -- this record is OK
ELSE
RAISE EXCEPTION 'User Score type is %, tried to set points = % and percentage = %', v_scoreType, NEW.points, NEW.percentage;
END IF;
Hopefully one of these ideas helps!