If I understand you correctly, you want Transaction to be related to Account via two relationships: debitAccount and creditAccount, yes? And you're wondering about creating inverse relationships.
In short, a relationship can only be the inverse of one other relationship. So you won't be able to create a relationship called, say, transactions that is the inverse of both debitAccount and creditAccount. Instead, you'll have to create two relationships, like debitTransactions and creditTransactions (I'm sure you'll think of more appropriate names for these...)
Now, since relationships are modeled as sets (specifically, NSSets), you can union the creditTransactions and debitTransactions relationships for a particular Account to get all transactions that account is involved with.
A (possibly better) alternative would be to introduce an intermediate entity, with a name like TransactionAccount that has a to-one relationship to both Account and Transaction as well as an attribute, like accountRole that identifies the account as being the debit or credit account relative to that particular transaction. You'd create inverse to-many relationships on both Transaction and Account with a name like transactionAccounts. That way, you could write something like this:
[account valueForKeyPath:@"transactionAccounts.transaction"]
to get all the transactions for a particular account. You could use an NSPredicate to filter the set to only transactions where the account was the debit/credit account.