I think you can just use swap - eg.
my_map["123"].swap(my_set)
provided clobbering my_set doesn't matter to you. This would swap the previous contents of my_map["123"] with my_set, and it's fast.
The indexing operator[] returns a reference to the contained set - therefore, you can manipulate it just like any other variable. If you want to add/remove individual values, you can just use insert() or erase() methods - eg. my_map["123"].insert(123).
The copying actually takes place when you assign a new set to the map - eg.
my_map["123"]=my_set
would create a copy of my_set. If you don't need to use the value of my_set later, you can use the swap() method, which will just shuffle pointers of the two sets. my_set will, however, have the contents of the previous my_map["123"].