Basically I write an application which copies/moves files from the local file system to a remote file system over some FTP-like protocol.
Would it be a good approach to encapsulate the protocol-specific bits inside of the file system service provider interface?
As far as I understand that would make my library work with other applications using the new IO API, right?
It seems that you are thinking of creating a RemoteFileSystemProvider class specifically for your remote file system. This class would mirror FileSystemProvider, providing similar functionality to access the remote file system you are using. If this is something that your project or team can get repeated use from, then it is worthwhile to mirror the FileSystemProvider object when creating your code. This allows anyone familiar with the java.nio.file package can easily understand how to use yours.
Keep in mind, however, that FileSystemProvider itself does not conform to any interfaces or extend any classes within the package. It is an entry point, and your class would also be an entry point. However, if you mimicked the method structure, you would be generating Channel objects to read and write, which would conform to the java.nio specifications that could be reused. This would allow any code which knows how to work with channels to be able to work with channels generated by your remote file system provider.
My first step in building something like this, however, would be to just build a package specifically for this remote file system. It would handle all of the communication and implement the basic functions like getUploadChannel, getDownloadChannel, renameRemoteFile, copyRemoteFile, deleteRemoteFile, and whatever other functionality is necessary. This will give you a good common-sense interface defined for this specific file system and its functionality. This package could be used in any context to just implement a connection to this file system. If you make sure that this object uses channels to upload and download files, then it will be ready to integrate with any API which uses java.nio.
Only after this was completed, tested, and working, would I consider which interface I'd like to mimic or implement for delivery to the rest of the team. This will ensure that any changes to the remote file system protocol or to the java API will have minimal impact on the overall system.
I would get down and dirty and use the FTP-like protocol. The trouble with implementing low-level APIs over high-level APIs is that you disguise all the true costs,and make things look easy that are in fact extremely hard, or expensive. Consider seek() for example.