Security plan for business critical information and service accessibility

Question

Hi,

I'm involved in creating a web based business solution. The idea is that the customers will use it, get their business processes and information into one place and also receive added business value by inter-system communication. In short they will use it as a core tool in their daily work and will depend highly upon it.

One problem in need of a solution is how to get this web system secure enough to be an alternative which both we and the customers will find satisfactory. I am looking for good advice from others who have been or are in the same situation.

In our specific scenario we're currently looking at using Java SE 6, Tomcat (as a Servlet container, needed as we will use Wicket), Hibernate (to interact with our database) and MySQL (as DBMS).

I think the problem and advice will be of interest for other technology users as well. As many of the issues are general ones regarding HDD failure, network accessibility and other things.

Feel free to give any advice you have! I still provide some questions and thoughts to get us going:

• The system needs to be reachable through the Internet. What should we think about when deciding on how to host it? (i.e. do we need our web host to have multiple physical paths connecting them to the Internet and similar questions.)
• Are there check lists for these kinds of things? Maybe ISO standards or some other way of seeing that we are on the right track by looking through an article/check list/academic paper/book?
• Later in the project we think it would be a good idea to get someone involved who has extensive experience in the field. In that case we're not looking for a normal web developer. It is likely that more consulting firms will tell us they are capable of providing this expertise then there actually are. Any tips on how we will get in contact with the right people? (We're based in Scandinavia, so it would be preferable to find someone there.)
• How high up time is good enough? 99.99% seems like a reasonable goal. But any downtime might result in loss of business for our customers.
• How do we guarantee that each customer only will be able to access its own data? As the system will be able to access it's own database, it seems hard. A proper development process, involving lots of testing, is really all we have regarding user privileges.
• How do we deal with HDD failures? Is RAID 5 in combination with a daily incremental backup and a weekly full backup enough? Or would you go for RAID 6?
• If one server is enough to serve the clients. Would you still use a cluster? (I would think so.) And in that case, how many nodes would you have in the cluster?
• Which backup strategy would you use?
• Do you think hosting the system in a computer cloud is a good alternative? (i.e. as provided by Amazon, Google or others.)
• Would you use hard disk encryption? And if so, which kind? (One clarification: Yes it's only good if someone steals the hard disk, but that's still added security and may prevent (physical) intruders access to vital client business data.)
• Is providing the customer with a way to do their own backups as well a good alternative? These customers won't be technically oriented. So in that case downloading the information in a ZIP archive containing Microsoft Office files might be a good way?
• How would you monitor the solution?
• Which of these things do you think we should do in house and which should be out sourced? We will develop the core system our self's, of course.
• If you feel that the system is secure, as a technical person. How do you convince a non technical person that it's safe and secure?

Thank you for your time! I hope you have some input to share. More questions might be added later.

Answer 1

Hi,

I too make web apps in my personal time and job so I can understand why you ask the above questions. While at work none of the above issues are discussed, I pay a lot of attention to these things in my personal work. I can't answer all your questions, but for the ones I can, I will say this:

How would you monitor the solution? I am developing an ASP.NET web app, and so I'm using performance monitors, logging, tracing (only for dev though), health monitoring, and a custom Sql Server performance monitoring system which I'm writing (although before this gets rolled out, I will use AppManager by AdventNet).

Which of these things do you think we should do in house and which should be out sourced? We will develop the core system our self's, of course.

This is an excellent question as I have the same decision to make, all the time. I would outsource what doesn't lie in my skillset (so if I need to do 3d work, outsource it). Also, stick to the strengths of your team, concentrate on business functionality as this is what will please/draw the users (I say this from a commercial point of view for a home-user facing web app), and outsource the critical internal systems like backup, monitoring, logging, etc (I haven't outsourced this due to the cost/financial resources I have available, and I make stuff so I can learn - thus code it myself even though this is the long approach - but I enjoy it).

Are there check lists for these kinds of things? Maybe ISO standards or some other way of seeing that we are on the right track by looking through an article/check list/academic paper/book?

You say you are using Java/MySQL/Hibernate, but Microsoft have some excellent guides on web application security and generally building scalable and secure applications (both web and windows based). Look at Microsoft Patterns and Practises.

Do you think hosting the system in a computer cloud is a good alternative? (i.e. as provided by Amazon, Google or others.)

Yes. This is cheap, effective, and will take stress off your internal architecture (I'm thinking of how you can use CDNs and Amazon EC2 to store static files). I say static files, so you can store content which doesn't change often, and which can be cached.

If you feel that the system is secure, as a technical person. How do you convince a non technical person that it's safe and secure?

Ask them to cause a breach.

How do we guarantee that each customer only will be able to access its own data? As the system will be able to access it's own database, it seems hard. A proper development process, involving lots of testing, is really all we have regarding user privileges.

I am not sure I understand what this question is asking, but you can use login systems, membership/roles (this is ASP.NET only though), and stored procedures to ensure that a user can only ever see his or her own data and not anyone elses.

I too would be interested in answers to the rest of the questions.

Answer 2

disclaimer: I am not a network engineer. I skipped the questions I felt I couldn't answer

The system needs to be reachable through the Internet. What should we think about when deciding on how to host it? (i.e. do we need our web host to have multiple physical paths connecting them to the Internet and similar questions.)

I would scale up as needed. In your first stage, you're probably not going to need redundant everything. Once people rely on you for their business needs, upgrade the shakiest parts of your network. If your internet is unreliable, get a secondary pipe. If your switches have gone down, double them up.

How high up time is good enough? 99.99% seems like a reasonable goal. But any downtime might result in loss of business for our customers.

The Nines. Don't pay more for uptime than you will lose for downtime. If you'll lose 50K for going down unexpectantly for an hour, you should be paying around 50K to keep downtime under an hour. 4 Nines is 52 minutes of downtime a year. Is that reasonable for you? Five nines is extremely difficult. Four nines for starting out is probably more difficult. I work for a financial, and we run about 98% to 99%, with about 50-80% of the downtime planned. We get by well.

How do we guarantee that each customer only will be able to access its own data? As the system will be able to access it's own database, it seems hard. A proper development process, involving lots of testing, is really all we have regarding user privileges.

We have the same problem. Good developers, proper development practices, automated testing, manual testing, development tools (e.g. attributes decorating proprietary data that raise warnings when the object is used in a public context), code scanners (e.g. looking for procs and code that use tables with a proprietary indicator without referencing it), code reviews, and so on.

Which backup strategy would you use?

Best I can reasonably afford. If possible, I'd say hourly for 24 hours, Daily for 3 weeks, Weekly for infinity.

Do you think hosting the system in a computer cloud is a good alternative? (i.e. as provided by Amazon, Google or others.)

Joel likes to say that your core business should not be outsourced. If your core business is storing documents, maybe no. If your core business is ancillary to the documents, seems reasonable.

Would you use hard disk encryption? And if so, which kind?

This would only help if someone stole your hard disk, not if they hacked your system. You're looking at encryption incorrectly. You dont want encryption for encryption's sake - you want it where it will actually help. HTTPS is good encryption. Maybe (-maybe-) encrypt each document on a document-by-document basis and require a passkey from the user to decrypt it.

Is providing the customer with a way to do their own backups as well a good alternative? These customers won't be technically oriented. So in that case downloading the information in a ZIP archive containing Microsoft Office files might be a good way?

That's a good feature to have no matter what. Billing it as backup is reasonable, but if you really want to provide their netops with a backup solution it will need to be more robust than that.

How would you monitor the solution?

SMS alerts to my blackberry, sanity checks on usage, someone on-call 24/7 with the ability to troubleshoot the system.

If you feel that the system is secure, as a technical person. How do you convince a non technical person that it's safe and secure?

A non-technical person? Put padlocks all over the website. Alternatively, convince their technical friend it's secure (by letting them see your security), and have the friend vouch to the non-techie.

Answer 3

The system needs to be reachable through the Internet. What should we think about when deciding on how to host it?

I'd consider a few things. First, your hosting provider needs to provide multiple, independent, network paths from the hosting location to the Internet. Second, what level of physical security are you willing to invest in for your machine(s)? Normally, at least in the UK, you (mostly) rent co-lo space by the U (height unit) and that means your machines will be accessible to anyone working in the same rack (maybe even the same machine room). If you rent whole racks, it MAY be possible to have your own keys.

How high up time is good enough? 99.99% seems like a reasonable goal. But any downtime might result in loss of business for our customers.

First, consider when this uptime needs to be present. 99.99% sounds good, but if the customers only need access Mon-Fri, 08:00-20:00, you can have just under 50% availability, with no lessening of customer satisfaction. Also consider that you will have to do maintenance (database checks, backups and other things) that MAY affect application availability, so I would make sure to include that in the service level guarantee (either as "X% availability in the ... window" or "X% availability outside announced, unplanned outages").

If one server is enough to serve the clients. Would you still use a cluster? (I would think so.) And in that case, how many nodes would you have in the cluster?

If one server is good enough for the foreseeable future, I wouldn't, personally, bother with a cluster, but I'd make sure to have identical machine(s) on the co-lo site, powered off, so the outage of one machine has minimal time-to-fix.

Do you think hosting the system in a computer cloud is a good alternative? (i.e. as provided by Amazon, Google or others.)

What guarantees do you have as regards uptime, data privacy and similar from a data cloud provider? You cannot (or, at least, should not) guarantee anything more than that to your customers. Also, if you can host your machine(s) in Scandinavia, you may have a network advantage over someone providing a similar service from a compute cloud hosted in the US. I honestly don't know if any of the compute cloud providers have servers in Scandinavia, but they may be happy to answer that question.

Which of these things do you think we should do in house and which should be out sourced? We will develop the core system our self's, of course.

"That depends". The more you do in-house, the less you have to pay to other outfits. On the other hand, if you can find good providers of peripheral services (data centre design, maybe basic machine build, maybe service monitoring, maybe security monitoring, maybe turn up X days a week to change backup tapes and bring old tapes to a safe location). One upside about doing it in-house is that it may act as a motivator for your staff. On the other hand, the more you do in-house, the more exposed you are, if one or more of your employees gets hit by a bus. So, for anything you do in-house, make sure you have at least two

Answer 4

The system needs to be reachable through the Internet. What should we think about when deciding on how to host it? (i.e. do we need our web host to have multiple physical paths connecting them to the Internet and similar questions.)

If you want 99% uptime, then yes you will. I would look at a company called Server Vault.

Are there check lists for these kinds of things? Maybe ISO standards or some other way of seeing that we are on the right track by looking through an article/check list/academic paper/book?

Official checklists, not that I have seen. Normally it means going through and having policies in places for each disaster and a mitigation.

Later in the project we think it would be a good idea to get someone involved who has extensive experience in the field. In that case we're not looking for a normal web developer. It is likely that more consulting firms will tell us they are capable of providing this expertise then there actually are. Any tips on how we will get in contact with the right people? (We're based in Scandinavia, so it would be preferable to find someone there.)

I would check with Internet Security Solutions or At Stake. I believe they are located all over the world.

How high up time is good enough? 99.99% seems like a reasonable goal. But any downtime might result in loss of business for our customers.

If you really want your customers to have their business rely on what you build, then down time really isn't acceptable. You can achieve this by having clustered servers and DR sites. At this point this might be feasible for you, so I would say having a maintence window maybe for a couple hours a month would be acceptable. You're going to have to check where your customers are located as having down time in the middle of the night for you may well be during the middle of the day for your customers.

How do we guarantee that each customer only will be able to access

its own data? As the system will be able to access it's own database, it seems hard. A proper development process, involving lots of testing, is really all we have regarding user privileges.

Lots of testing. I would suggest frequent penetration tests to make sure you don't leave a hole somewhere.

How do we deal with HDD failures? Is RAID 5 in combination with a daily incremental backup and a weekly full backup enough? Or would you go for RAID 6?

I would go with some type of Raid and multiple servers. You'll need some sort of monitoring to alert you if a server goes down. Depending on where you host it, they may have some sort of monitoring solution you can hire them for, so they will be responsible for letting you know if something goes offline. I would reccomend using Nagios for server montoring.

If one server is enough to serve the clients. Would you still use a cluster? (I would think so.) And in that case, how many nodes would you have in the cluster?

Yes you never know what might happen.

   Which backup strategy would you use?

A clustered solution with a DR site, log shipping (I know sql server I know there is a process to keep multiple db servers in sync), and nightly backups. All backups should be shipped offsite and kept for at least 7 days. They should also be encrypted to ensure data safety.

Do you think hosting the system in a computer cloud is a good alternative? (i.e. as provided by Amazon, Google or others.)

I don't know if I would trust them yet, and I have heard they are very expensive. For a mission critical system, I go with what I know and use proven technologies. That's my opinion though. They may work fine.

Would you use hard disk encryption? And if so, which kind? (One clarification: Yes it's only good if someone steals the hard disk, but that's still added security and may prevent (physical) intruders access to vital client business data.)

Probably not. It will slow the servers down too much. I would rather go with a hosting service that I can trust to keep the servers secure.

Is providing the customer with a way to do their own backups as well a good alternative? These customers won't be technically oriented. So in that case downloading the information in a ZIP archive containing Microsoft Office files might be a good way?

It might be nice to offer the option, but if you can show them a comprehensive backup strategy, I would say that would suffice.

How would you monitor the solution?

Something like Nagios, and hire the hosting service to watch the server health. For 100% uptime it really needs to be in something like a NOC where someone is always there watching the server monitors.

Which of these things do you think we should do in house and which should be out sourced? We will develop the core system our self's, of course.

Do the things you are comfortable with and compentent at. It sounds like programming and testing. I would hire people to do server setup, hosting and monitoring. The key to this is don't go with the lowest bidder. Go with the people who will do a good job.

If you feel that the system is secure, as a technical person. How do you convince a non technical person that it's safe and secure?

You have a written set of policies and procedures that you follow. You have a list of all vendors you use with what purchase from them and how they are configured to run. You make sure that all vendors you contract from are reputable and have a proven track record.