So, you have a project with a continually changing team size and your boss wants you to give him an accurate estimate of how long it will take? You can do this, as long as you keep in mind the difference between accurate and precise. Your precision will depend largely on the number of items and how granular (decomposed) each item is; the more items you have the more the Law of Large Numbers works for you, averaging out over- and underestimates.
Your accuracy is a function of confidence. Note that estimates aren't single-point values, they're a range with numbers with a percentage of confidence. For instance, a proper estimate wouldn't be "2 weeks" it would be "50% confidence of 2 weeks, 80% confidence of 4 weeks."
If I were the person assigned with the unenviable task of providing an estimate to completion for a project being managed as arbitrarily as in the original post, I'd try to figure out a range based upon the minimum number of folks assigned (e.g., "48 to 66 weeks given 2 developers [50% to 80% confident]"), and a range associated with the average number of folks assigned (e.g., "25 to 45 weeks with 5 developers [50% to 80% confident]"), and use the low figure from the average number along with the high figure from the minimum number (e.g., "25 to 66 weeks given anywhere from 2 to 5 developers [50% to 80% confident]"), and even then I'd put a disclaimer on it ("plus 10% for the lost time due to context switching").
Better yet, I'd explain exactly why this arrangement was, to be polite, sub-optimal, and why multi-tasking is a primary signpost on the road to project Hell.
As someone else suggested, changing the workflow from iteration-based to flow-based (Kanban) might well be a good strategy. With Kanban you handle changing project priorities by changing the priority of items in the backlog; once an item has been grabbed by the team it is generally finished (flows all of the way through the workflow, stakeholders aren't allowed to disrupt the team by screwing around with work-in-progress). I've used Kanban for sustained engineering projects and it worked very well. Re how it would help with estimates, the key to continuous flow is to try to have each work item be roughly the same size (1x, 2x, 3x, not 10x, 20x, 100x). You should track movement of items through the workflow by tracking dates of process state changes, e.g., Queue 1/15, Design 1/22, Dev 1/24, Test 2/4, Integrate 2/7, etc., and then generating a cumulative flow diagram regularly to evaluate the time-in-state durations over time. Working out how long the project should take given that you know the size of each item and the time through the workflow for items is a trivial computational exercise left to the reader. (The more interesting question is how to spot constraints... and then how to remove them. Hint: look for long times in states, because work piles up in front of constraints.)