It seems to me that a good unit for a Story Point would be ideal man-hour, not man-day.
This phrase sounds really, really strange, and not true. Where did you read that there is a correlation between Story Points and ideal man-day? Ideal man-days were maybe used in the early days of Scrum but, to me, Story Points (SPs) are a different thing...
Story Points are a way to to quantify the relative effort associated with a particular Product Backlog Item (PBI) which is composed of multiple tasks. Some teams use numeric sizing (i.e. a scale of 1 to 10) to estimate the "size" of a PBI, others use t-shirt sizes (XS, S, M, L, XL, XXL, XXXL), some use the Fibonacci sequence (1, 2, 3, 5, 8, 13, 21, 34, etc). And by the way, did you notice that SP are unit-less?
If I used days, most of my issues would have estimates 1/2 or 1.
So what? That would just mean that you have small PBIs, which is not a bad thing (at least not for the most important one). But don't forget that there are theoretically two level of estimation in Scrum: the Product Backlog level, in points, and the Sprint Backlog level, in hours. As I mentioned in the previous paragraph, PBI are composed of tasks and they should be split into tasks during the second part of the Sprint Planning Meeting. And tasks are then estimated in hours and the 16h rule applies here: a task should not exceed 16h. If it does, it is too big and should be split into smaller tasks (because we are too bad at estimating big things).
Do you have any idea, why the use of ideal man-days is mentioned most often in the Scrum literature?
This is outdated anyway. Things might change in the future but the current consensus is to estimate in unit-less points. Points are entirely decorrelated from any time unit and this is intentional to avoid any mapping with real world unit, work capacity should be measured with the velocity (the amount of points a team can achieve in one iteration).