Industry Reports on Source Control Tools

Question

Hi,

I'm looking for independent industry reports that compare and contrast the various source control tools out there. In particular, I care about Clearcase vs Sourcesafe vs SVN, but if the report includes other SCM systems that's fine.

I need this for a client who wants to get a feel on exactly what they stand to gain switching to SVN (yes, from Clearcase and VSS). In other words, something I can use to sell it to their business.

I'm hoping some case studies have been done on developer productivity with these tools and resultant reports made freely available.

Thanks, Kent

Answer 1

I looked for something similar when I first started working at a company that did not have source control (yikes). The person in charge of development at the time would basically only consider Microsoft stuff, and the only other person on the small dev team that had any SCC experience only had used SourceSafe.

I've had a ton of experience with SVN, so admittedly I'm quite biased, but I really did try to do an evaluation of SVN vs VSS vs TFS. I really had trouble finding anything positive about VSS, other than by people who were comparing it to not using source control at all.

---

Sourcesafe

• Visual SourceSafe: Microsoft's Source Destruction System

  • From article:
    • SourceSafe lacks usable branching support [..] SourceSafe's merge support is tightly integrated with checking in, making it difficult to examine differences and test the proposed merge before checking it into the tree. With this weak level of support, it's easy to check non-functioning code into the revision control system.
    • When updating your local workspace to match the server, files which were deleted on the server should brought to your attention. (Or deleted, since the old version can be retrieved from the revision control system.) Failure to do so risks out of date files being used in your project, often causing problems. [..] SourceSafe fails to delete the out of date file or provide any warning.
    • SourceSafe's fundamental design assumes that clients are trustworthy, always function correctly, and that nothing interferes with the communication causing corrupted data. As a result, SourceSafe is fragile and untrustworthy. I have worked with SourceSafe at three different jobs. In each case, eventually the SourceSafe database became corrupted. Data has been corrupted, work has been lost, time has been wasted on the problem.

• Visual SourceSafe Version Control: Unsafe at any Speed?

  • From article:
    • SourceSafe seems to work reasonably when one uses a simple cycle of creating and then developing source files, checking the files in at given milestones, labeling the project, and so on. However, many simple actions cause the software to fail in various subtle (and sometimes unsubtle) ways. ''A file that has been deleted may easily be purged from the database, preventing previous versions of the project from being rebuilt. ''
    • A user could reasonably expect that the Destroy command deletes the current working version of a file from the user's hard drive. A user might also reasonably expect that the Destroy command deletes the file from the current version of the project. The user might not expect that the file is deleted from all previous versions of the project.

• CodingHorror: Source Control: Anything But SourceSafe

  • From article:
    • SourceSafe was a perfectly adequate source control system in the late 90's. Unfortunately, SourceSafe was never updated architecturally to reflect modern source control practices. Even the latest version, SourceSafe 2005, absolutely reeks of 1999.
    • SourceSafe gives you the illusion of safety and control, while exposing your project to risk.
    • SourceSafe teaches developers bad habits: avoid branching, exclusive locks, easy permanent deletions.

• http://www.wadhome.org/svn%5Fvs%5Fvss.html

• From http://svn.haxx.se/users/archive-2006-11/0242.shtml:

  • VSS by default uses locking when working with files. If a user checks out (locks) a file, then leaves the company or goes on vacation for 2 weeks, an administrator has to manually unlock the file if someone else needs to work on it, and when that other person comes back, you've got a merge mess on your hands. Merging is not terribly good from what I recall.

Short VSS vs SVN (taken from http://better-scm.berlios.de/comparison/comparison.html):

• VSS is not atomic - if network goes down, power goes out (on server OR client), etc, the repository is corrupted. SVN is atomic, either the entire change gets in, or none of it does
• VSS locks files - multiple people cannot work on the same file at the same time
• Changes in VSS are file-specific. SVN can commit changes in multiple files as one "changeset".
• VSS does not track per-line changes. With SVN it is possible to see who last changed a line, and in which revision ("svn blame")

Subversion

Very simple source code management system. Tracks changes, uses a update-merge-commit model, which allows multiple developers to work on the same file at the same time, and subversion automatically merges their changes (when possible). No "magic" happens.

Integration to Windows/Explorer via TortoiseSVN. Integration to Visual Studio via VisualSVN ($50/developer) - which is really just a front-end to TortoiseSVN.

Integrates with many 3rd party tools, eg:

• Trac for bug tracking
  • Tons of plugins/addons for this at http://trac-hacks.org
• CruiseControl.NET for continuous integration (automated builds)

Svn Cons:

• No built-in merge tracking (keeping track of which changesets have been merged from eg, development to stable branch)
  • Possible with svnmerge / svnmergegui
  • As of SVN 1.5, there is built-in merge tracking
• Missing "shelve" feature
  • Allows someone to work on something, and "shelve" it into the repository (without making it part of the project) where other developers can pick it up.
  • TFS has this
  • Can approximate this feature using either patches, or by creating a separate branch.

Team Foundation Server (Visual Studio Team System)

Uses SQL server as a backend (so it's reliable). Has a built-in set of tools for bug tracking, project planning, continuous integration.

• Retail pricing:
  • $469.98 for workgroup edition (5 user limit)
  • $2,499.98 for full edition

• Licensing may be more complex than this: Licensing whitepaper

  • It seems a CAL (client access license) is needed for every user of the system that directly accesses TFS. For example, even to allow a tech support employee to directly file a bug would require a CAL for that user.
  • Visual Studio Team Edition comes with a CAL - but separately they are ~$500 list price (according to what I've seen on forums)
  • This may be changing in TFS 2008

• Migrating from VSS to TFS : http://manicprogrammer.com/cs/files/folders/st%5Fjean/entry1118.aspx

• Performance issues: http://www.cornetdesign.com/2007/05/vststfs-performance.html

• http://weblogs.asp.net/rosherove/archive/2007/04/29/tfs-or-not-being-a-perfectionist-is-a-realistic-world.aspx
• http://jeremydmiller.blogspot.com/2005/07/impressions-from-vsts-talk-last-night.html
  • From article:
    • VSTS is clearly aimed at very large companies using laborious, high ceremony processes--everything that my employer is not. I think process automation is a great thing, but if your process is so complex that nobody can know the whole, I think you've got some serious issues.
    • The one piece of the presentation where I thought Menegay was completely off his rocker was the idea that VSTS became a directed workflow tool to tell everybody (expecially developers) exactly what they should be doing. As a tracking tool for project record keeping, I think VSTS is great. Using something like VSTS as your primary communication channel is absurd. Turning developers into mindless zombies doing exactly what the VSTS workflow tells them to do sounds like a recipe for disaster to me. Not to mention a steep drop off in developer retention.
• http://codebetter.com/blogs/eric.wise/archive/2007/05/31/bye-bye-team-foundation-server.aspx
  • From article:
    • I really tried to give team foundation server a fair shake. My opinion on it is as follows: 1. It's a generally better source control system than source safe. 2. It's really expensive. 3. It's a pain in the ass to administrate. 4. It's a pain in the ass to troubleshoot.
• By the sounds of it, merge/branch support is not quite as good as subversion, because they tried to keep it "familiar" for sourcesafe users. This is purely anecdotal.

---

My super-quick summary:

• VSS sucks
• TFS seems very usable/capable, but quite expensive as soon as your team grows past 5 people
• SVN is usable and well-proven, and has no costs

What it came down to for us - the possible benefits of TFS were not worth the huge extra cost.

For what it's worth, it's now a couple years later since I wrote this. The guy who used to be in charge is no longer with the company, I'm now the development manager, and our team is 3 times as big. We use a combination of SVN, VisualSVN + TortoiseSVN, and Trac. I don't think anyone in dev could imagine not using these tools anymore. Everyone was able to pick up SVN very quickly, with maybe the exception of merging between branches, which some are still not confident about doing on their own.

---

Jeff Atwood (creator of stackoverflow) discusses SVN a bit on one of his and Joel's podcasts: http://blog.stackoverflow.com/2008/06/podcast-10/

Answer 2

There are SO many places to compare one to another, but they almost always are biased toward one system.

Here are some comparisons I've used in the past.
better-scm
A dynamic comparison site
wikipedia

The Wikipedia site is actually quite thorough and easy to read.

Answer 3

Forrester Research puts out a report every few years. The 2007 version is cited by IBM and CollabNet, the people behind Subversion.

Answer 4

we've been using SVN at my work for about a year (dictated from on-high). All I can say is, if you want to do merging, stay away from SVN with a long pole. Seriously. It's detrimental. For basic source control it's fine. With merging, it becomes a nightmare. And don't tell me we are using it wrong. We've tried everything. It's just not developed to work properly for merging. But if you decide to do it, I wish you better luck than we've had.