See also How does a file with Chinese characters know how many bytes to use per character? - no doubt, there are other SO questions that would also help.
In UTF-8, you get the following types of bytes:
Binary Hex Comments
0xxxxxxx 0x00..0x7F Only byte of a 1-byte character encoding
10xxxxxx 0x80..0xBF Continuation characters (1-3 continuation characters)
110xxxxx 0xC0..0xDF First byte of a 2-byte character encoding
1110xxxx 0xE0..0xEF First byte of a 3-byte character encoding
11110xxx 0xF0..0xF4 First byte of a 4-byte character encoding
(The last line looks as if it should read 0xF0..0xF7; however, the 21-bit range of Unicode means that the maximum valid value is 0xF4; values 0xF5..0xF7 cannot occur in valid UTF-8.)
Looking at whether a particular sequence of bytes is valid UTF-8 means you need to think about:
- Continuation characters appearing where not expected
- Non-continuation characters appearing where a continuation character expected
- Incomplete characters at end of string (variation of 'continuation character expected')
- Non-minimal sequences
- UTF-16 surrogates
In valid UTF-8, the bytes 0xF5..0xFF cannot occur.
Non-minimal sequences
There are multiple possible representations for some characters. For example, the Unicode character U+0000 (ASCII NUL) could be represented by:
0x00
0xC0 0x00
0xE0 0x00 0x00
0xF0 0x00 0x00 0x00
However, the Unicode standard clearly states that the last three alternatives are not acceptable because they are not minimal. It so happens that the bytes 0xC0 and 0xC1 can never appear in valid UTF-8 because the only characters that could be encoded by those are minimally encoded as single byte characters in the range 0x00..0x7F.
UTF-16 Surrogates
Within the Basic Multi-lingual Plane (BMP), the Unicode values U+D800 - U+DFFF are reserved for UTF-16 surrogates and cannot appear in valid UTF-8.
Bad Data
So, your BAD data should contain samples violating these various prescriptions.
- Continuation character not preceded by one of the initial byte values
- Multi-character initial bytes not followed by enough continuation characters
- Non-minimal multi-byte characters
- UTF-16 surrogates
- Invalid bytes (0xC0, 0xC1, 0xF5..0xFF).
Note that a byte-order mark cannot appear unencoded in UTF-8 - the bytes 0xFF and 0xFE are not permitted in valid UTF-8. An encoded zero-width non-breaking space (U+FFFE) can appear in a UTF-8 file as 0xEF 0xBB 0xBF. But the BOM is completely superfluous in UTF-8.
There are also some non-characters. U+FFFF is one such - it cannot appear in Unicode, and hence should not appear in UTF-8 encoded data either.