create table setmember (setid int, setmemberid int)
create unique nonclustered index idx_setmember_idx1 on setmember (setid, setmemberid)
Let me assume a magic number (-1 or 999999999) for "all".
This will be highly performant for both querying on a per set basis, and update insertion via the nonclustered index. Uniqueness enforces no repetition of entries. It is problematic to enforce either "all" or else multiple set members as a constraint, but there are diminishing returns though it could be done in a trigger.
Additionally add
create nonclustered index idx_setmember_idx2 on setmember (setmemberid, setid)
to allow for efficient reverse lookup queries.
If you use array types, you may be unable to implement efficient reverse lookup.
Note all the SQL above is ANSI compliant.