You could achieve this through a post-receive hook. This example hook just echos back possible details that could be logged to the pushing user, but the details could be logged via syslog or a dedicated loggin mechanism of some kind.
# sample logging post-receive hook
echo This hook logs back to the pusher, but could append to syslog or something.
echo push from user $LOGNAME at $(date), ssh client details: $SSH_CLIENT
echo refs updated:
echo -------------
cat
echo -------------
For the SSH_CLIENT details to be accurate, you probably want to ensure that users don't have direct shell access as otherwise they could get a shell via ssh, then spoof their LOGNAME and SSH_CLIENT info, although such fudging would be malicious and giving malicious users push access to a git repository is inherently dangerous. There are more reliable ways to determine the users identity but if the user has control over there login scripts then, as the hook is run as the logged in user, there is always the possibility that they can manipulate it in a way to subvert how the hook works.
This way you get the from and to commit SHA1 for each branch changed. In order to ensure that you can always easily examine the history you probably also want to make sure that the config variables receive.denyDeletes and received.denyNonFastForwards are set to true. Again, to deny fiddling with these configs, you want to avoid users having shell access to the repository.
If you don't mind about non-fast forwards or branch deletes, you can still access old commits that haven't been pruned through the reflogs, so long as these are enabled via the core.logAllRefUpdates.