My WCF server exposes database access remotely and ensures authentication. Typically, users download and upload large documents. The remote connection turns out to be quite slow.
Are there any tips and tricks for optimizing the service to handle those types of communication effectively?
This will depend on the protocol you are using. For example if you use XML over HTTP you could use MTOM to reduce the size of the data and which is optimized for transferring files over HTTP.
For large documents, binary or streaming transfer is a good idea to optimize performance.
Both binary and streaming transfer do not require Base64 encoding, which means that there is no 4:3 ratio for the size of transferred data.
Streaming also reduces memory footprint (especially when using FILESTREAM in Microsoft SQL, which is probably your case if you store large documents in database), so may improve performance a bit more.