There's not really such a thing as a “Unicode version” of a font(*). “Arial Unicode” is a misleading name: it's not materially different to normal “Arial”, it just has some more characters in it. It does not contain usable glyphs for every single one of the tens of thousands of characters defined so far, and indeed there is no one OS standard font that does.
The significant question is merely whether the characters you want to use have glyphs in the default fonts of commonly-deployed operating systems. You need to look to look at font support for particular characters you wish to use on an individual basis.
The character U+0360 Combining Double Tilde you mentioned is not really ‘advanced punctuation’, it's an curious and rarely-used diacritical mark for phonetics work. So it's not really surprising that font support for it is poor. On the other hand, Stack Overflow can get away with using U+25CF Black Circle (●) because lots of fonts have it. Some of the other characters from the Geometric Shapes block such as U+25B2 Black Up-pointing Triangle (▲) are also pretty common.
fileformat.info has a list of common fonts that support each character, so you can check there to get a feel of how widely supported a symbol is, and whether the default OS fonts you recognise are present, before using it as a replacement for an image. For example U+25CF is in many fonts, but U+0360 isn't that well-supported: none of the default Windows install fonts are there, and the ‘Libertine’ font renders it badly wrong.
(*: OK, there is sort of such a thing as a Unicode font, in that a font's internal character lookup tables may be denominated in Unicode or some other character set. However this makes no practical difference as the application will always be addressing it as Unicode; the OS will do the conversion on lookup transparently.)