I'm just wondering, if a thread is in a critical section, can it be preempted?
If Thread A preempted, and so the mutex lock is stuck with Thread A, what can be done about this?
Of course it can be preempted. Otherwise how the other threads could try to enter that critical section, if the only thread that is allowed to run within the process is the thread that owns the critical section?
Thread B in your example will wait until thread A is rescheduled and is finished with the crtical section. No surprise here. And if thread A, while in a critical section, also waits for a mutex owned by thead B, then it's a deadlock which you must resolve by revising your logic.
Suppose thread A is preempted by higher priority thread C. Now suppose thread B is in fact higher priority than C. If B becomes runnable, you have a classic case of priority inversion; Thread B (high priority) is stuck waiting for a resource held by Thread A (low priority). One cure for this is called priority inheritance.
With priority inheritance, when B blocks for the resource held by A (the critical section), thread A temporarily 'inherits' the priority of thread B. This allows A to preempt that bothersome middle-priority thread C, and when A is done with the resource, A goes back to its original priority. This gets A out of B's way, so to speak, eliminating the dead-lock.