100% untested, and off the top of my head, but you can give something like this a go. If I have a chance to test tonight I'll update the post, but there's a bottle of wine open for dinner and it's Friday night... :)
WITH CTE AS
(
SELECT
ColumnB,
ColumnA,
ROW_NUMBER() OVER (ORDER BY ColumnB) AS RowNumber
FROM
dbo.SomeTable
)
SELECT
CASE WHEN RowNumber <= 5 THEN ColumnB ELSE 'Other' END AS ColumnB,
SUM(ColumnA) AS ColumnA
FROM
CTE
GROUP BY
CASE WHEN RowNumber <= 5 THEN ColumnB ELSE 'Other' END
ORDER BY
MIN(RowNumber)
EDIT: Looks like this worked after a couple of silly syntax errors. I've corrected those, so it should work as listed above now. I can't speak to performance on a large data set though, but it's worth giving it a shot.