Firstly, you will need to modify the web.config where your SharePoint virtual directory resides. This is because you'll need to add SafeControl entries to trust the log4net assembly. You can update the web.config programmatically using the SPWebConfigModification class in a feature receiver. As you have to modify web.config anyway, you may want to consider including your log4net config inside and not set up an external log4net config.
However, if you'd still like to do this, it may work if you add the following to the web.config file:
<configuration ...>
...
<configSections>
<section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,log4net" />
</configSections>
<log4net configSource="log4Net.config">
...
</configuration>
The log4net.config file should then be able to live alongside your web.config. As Nat says, you could deploy this file as a solution package.
Assuming you are attempting to run a minimal trust, you will need to update your Code Access Security file to include the log4net assemblies as well. All of your custom SharePoint code should then automatically use your log4net configuration.