You can accomplish this by creating a computed column and put the unique index on that column.
ALTER TABLE MYTABLE
ADD COL2 AS (CASE WHEN COL1 IS NULL THEN CAST(ID AS NVARCHAR(255)) ELSE COL1 END)
CREATE UNIQUE INDEX UQ_COL2 ON MYTABLE (COL2)
This is assuming that ID is the PK of your table and COL1 is the "unique or null" column.
The computed column (COL2) will use the PK's value if your "unique" column is null.
There is still the possibility of collisions between the ID column and COL1 in the following example:
ID COL1 COL2
1 [NULL] 1
2 1 1
To get around this I usually create another computed column which stores whether the value in COL2 comes from the ID column or the COL1 column:
ALTER TABLE MYTABLE
ADD COL3 AS (CASE WHEN COL1 IS NULL THEN 1 ELSE 0 END)
The index should be changed to:
CREATE UNIQUE INDEX UQ_COL2 ON MYTABLE (COL2, COL3)
Now the index is on both computed columns COL2 and COL3 so there is no issue:
ID COL1 COL2 COL3
1 [NULL] 1 1
2 1 1 0