Sharepoint as a high volume information system

Question

I'm looking at designing some core information systems at a new company I'm working at (described one of my ideas here http://stackoverflow.com/questions/908705/workflow-system)

I've thought a bit more, and am strongly considering using sharepoint for a lot of the heavy lifting seeing as it comes with so much out of the box.

However, I'm not sure how it will handle the high volume of data we'll be throwing at it. I read the MS whitepaper (http://go.microsoft.com/fwlink/?LinkId=95450&clcid=0x409), and it says about 2000 items in a list is about the limit using traditional design methods.

But first a bit of info on my plan and data structures :

We have multiple clients. Each client has multiple applications. Each application will have multiple, ongoing jobs (or process runs).

Each application will store significant correspondence and documentation. Each job represents the processing of a data file on a single run, and stores information about the job such as the postscript file, postal manifests, etc.

Job volume will be about 50 - 100 a day. Each job will have a workflow, triggered by external programs. Then, say on a "job scheduler" page, production staff can schedule the jobs and perform custom actions on the job (written as plugins).

I was thinking the jobs would sit outside and accessed via the BDC, but I would still like them represented in sharepoint lists, to add in sharepoint functionality and reporting, and they'd be accessible in multiple places

e.g.

• Application portal - see jobs for application
• Production scheduler - see lists of upcoming jobs, assign to resources, trigger other functionality (e.g. copy print file to printer, produce mailing machine file)
• Invoicing view - view completed but uninvoiced jobs, export to accounting package
• Client view - client portal displays jobs, invoices, stock levels (from external warehouse system), documentation, change register / helpdesk

So basic info about the job would sit in the BDC, but then sharepoint would capture additional metadata about each job. Also, down the line we might put in more advanced workflows using WF or something like K2 blackpoint / blackpearl.

Is this feasible? Any resources you'd recommend to read to get up to speed?

Answer 1

Tough question Dane... I would like to know a little more about your design / vision before giving an opinion.

Based on what I read in your question I would not use SharePoint 2007 as a development platform for this application.

1) Development experience in SharePoint 2007 can be painful and unproductive at times.

• Hard to debug
• Steep learning curve

2) Easy to get in trouble with performance

• Data Layer is complex and can require expert SQL / SharePoint Admin skills to make platform scale.
• Content databases should not exceed 100 GB.

3) Deployment can be extremely difficult depending on what you are doing.

4) New version will be released in the next 12 months.

Just my .02.

Answer 2

SharePoint is probably quite a good fit for the UI side of things, though you'll need to think carefully about which parts are stored and modified in SharePoint lists and which parts are stored elsewhere. That's not so much a SharePoint issue as something you always have to deal with when you have multiple data sources.

I'd probably use a SharePoint list as the primary store for jobs, to avoid any sync issues and make editing easier. The volume of data shouldn't be an issue - just make sure you aren't trying to display 2000 items at once - it's the view, not the list itself that runs into performance issues on large numbers of items.

Answer 3

To use SharePoint, you should concentrate on what SharePoint is good at and what it is designed for.

SharePoint is a great collaboration portal, it is not so good as a simple high volume database. So...

You can setup a small site for each client and subsites for each job. The goal of the "job site" is to display (using a webpart perhaps) the relevant upcoming jobs, a list of job errors/exceptions and relevant team documentation on each job.

Separate sites can be created to give a particular "view" of the jobs. E.g an "Invoicing" site can be created to give a view again from BDC webparts of what is requiring invoicing.

https://iwsolve.partners.extranet.microsoft.com/SDPS/ may provide some help.

Don't try and store huge amounts of information in a SharePoint list, just because it may be possible to "tag" it with meta data. A database table is perfectly able to include columns supplying additional information if required.

Think about it this way. If you are creating 50-100 jobs a day, putting that data into a list pre-supposes your sites users are going to want to enter metadata on those jobs manually. I thought not, so create systems you need in order to get the metadata stored correctly at source, or store metadata about the "types" of jobs within a SharePoint list and allow SharePoint to match the job type with jobs in the BDC.

SharePoint will help you to integrate all your systems information together, but unfortunately it looks like you have a lot of work to do just planning what information should go where and how each type of use will view it.

Answer 4

Hi Dane,

Please take a look at this blog post I wrote on managing large SharePoint lists for better performance- it might offer a bit of an explanation for the 2,000 items issue, which is not actually a hard limit on the number of items in a list, as SharePoint will support up to 5 million items per list. One way around this would be to create and maintain different views that filter by an indexed field to show you different items, up to 2,000 at a time. Hope that helps.

Dina Ayoub Program Manager Windows SharePoint Services