Short answer: "Why do I have to include the directory of the header files of X in E?"... You shouldn't. Client's of Y should not have to know that Y depends on X.
Long answer: You need to include the header of X in E only if the interface (signatures) of Y makes use of things declared in the header of X. But if the header of Y were "properly constructed", then they would include the headers of X in the Y header itself and you wouldn't have to include the X header in E explicitly (including the Y header would automatically include the X header).
By "Properly constructed" I meant that if the signatures in Y1.h in Y depend on (say) X3.h and X7.h, then Y1.h should include those files (directly of indirectly). This way, any client of Y1.h would not have to know what it's dependencies are and have to include those dependencies separately as a result. As a simple test, a .cpp consisting of the following line should compile without issues:
#include "Y1.h"
A good practice is to #include "Y1.h" before including any other file in Y1.cpp. If Y1.h is missing dependencies, the compiler will let you know.