Being an amateur symfony developer myself and having worked with more experienced symfony devs, my impression is that this is largely dependent on both what best suits the application AND your preference to organising the code. Once you start working on a more complex application, these kinds of questions answer themselves to some extent, as you will begin to notice how the organisation of your projects affects the ease of working with it.
On a site I'm building currently, I have about 20 modules, with:
- modules that have single action & template pairs
- modules that have single actions serving multiple templates
- modules that have multiple actions serving multiple templates
In total, there are probably 40-50 actions.
Apart from organising your code in relation to your models, the module structure also has significance in relation to the Symfony configuration cascade. Because layouts and configuration files such as view.yml, security.yml, etc are most easily set at app & module levels, good structure helps you take advantage of the cascade in a straightforward way. The same applies to routing rules, depending on how you use these on your site.
I don't have enough insight to comment on performance -- perhaps someone else here does. Would be interested in hearing about that myself.