Your system call starts a separate process while Perl's own rmdir will call the native C function in the context of the Perl process. Both methods end up doing the same system calls, but opening a new process is less efficient.*
It's best practice to use Perl's own functions (such as rmdir): they are portable, and you don't have to worry about interpreting shell exit codes, escaping metacharacters in filenames to prevent security risks, etc.
*system will create an additional sh process if your command contains pipes, input/output redirection, && etc. This does not apply in your case, but can make the system option even slower.