If post-backs don't bother you, then just do what you mentioned on server side in response to button click event.
Have some container, add and remove components dynamically, attach your event handlers to events of newly created objects.
Wrap the controls you want to show/hide in a panel. Then in the click event handlers of the buttons set the Visible property of the panel to true or false as you need.
Beware that your + button will need to be outside the panel otherwise you'll never get them back again.
You could use postbacks and add your controls server-side, you could even use ajax to make it more seamless.
If you are going to go this way, remember to re-create your dynamic controls on each postback