I think what you're saying is that for any active question, the tuple (surveyDomain, surveyQuestion, surveyAnswer) must be unique?
Or in other words, survey:surveyanswer is 1:1 if the survey is active, even though survey:surveyanswer is set up to be 1:many.
If so, the answer is to change your table structure. Adding a nullable activeAnswerId column to survey will effectively make the relation 1:1; your existing constraint unique SurveyId (or unique SurveyId, SurvetDomainId) will suffice to enforce uniqueness.
Indeed, unless I'm misunderstanding, I'm surprised that Survey has a Question column; I'd expect Survey:Question to be 1:many (a survey has many questions) or even many:many, if a question can show up on more than one survey.
More generally, I suspect the reason that figuring out how to enforce the constraint is difficult and requires "heroics" like triggers or user defined functions, is a symptom of a schema that doesn't accurately model your problem domain.
OP comments:
no, you're missing it. Survey:Answer is 1:n. "Question" is the survey question – Tuple would be (SurveyDomain.SurveyDomainId, Survey.Answer)
You mean that for every domain, there's at most one answer? Again, looking at your schema, it's misleading at best. A SurveyDomain has many Surveys (each of which has a Question column) and a Survey has many Answers? (Schema)
But if the Survey's active bit is set, there should be only one Answer?
Is Survey a misnomer for Question?
It's really not clear what you're trying to model.
Again, if it's hard to add a constraint, that suggests that your model doesn't work.