The debugger won't step through units that don't have debug information, so the goal is to make the compiler omit debug information from the units you're not interested in.
Put your library units in a separate library project. That gives you the ability to have separate compilation settings for those units without affecting your project. Compile the library without debugging information enabled. Then remove those library units from your project. You can continue using them, but they won't belong to your project anymore.
An important aspect here is that the DCUs should reside in a separate directory from the source code. If the compiler finds DCUs and it happens to see the source code in the same folder, then it's liable to recompile that code when you really don't want it to. Set your projects' "DCU output folder" to something other than the default.
To really do it right, you can do what the VCL does and compile two different versions of your libraries. Compile one with debug information, and one without, and put the compiled files in different directories. Add the directory with the debug versions to your Delphi configuration; there should already be a folder listed there that contains the Delphi-provided debug DCUs.
When you set up two different versions, you allow yourself to choose whether you want to step into the library code. Simply toggle the "use debug DCUs" option in your project settings. Delphi will automatically add and remove the debug-version folder from the search path when you toggle that setting.
Note that even though you'll have a separate library project for your library units, you don't need to link to or distribute the DLL or package that that project generates. You can continue using the DCU files directly in your EXE project. You're only setting up the separate project so that you can select different compilation settings for those units. Add the library project's DCU output folder to your EXE project's search path, and you can continue using the units directly without any need to distribute the library project's DLL or package.
The IDE may try to add new directories to the search path automatically. Don't stand for that. If there's a source directory there that the IDE added for you and you don't want it there, feel free to remove it. The IDE is just trying to be helpful, but it doesn't know about your plan to have separate source and compiled folders.