What percentage of my time will be spent in user input verfication during web development?

Question

I'm new to developing things on the web. So far, I'm spending a lot of time (50% or so) to try and prevent bad people from putting things like sql injection into my input forms and validating it server side. Is this normal?

Answer 1

No. It is not normal. Maybe you need to:

• Use the rights components to avoid SQL Injection (PreparedStatements in Java)
• Create a component that "filters" messages coming from user (a servlet Filter in Java).

Any modern language has support for both things.

Kind Regards

Answer 2

I'm glad you're taking care to protect yourself. Too many don't.

However as others have said, a better choice of architecture will make your problems go away. Using prepared statements (most languages should have support for that) will make SQL injection attacks go away. Plus with many databases they will result in significantly better performance. Handling cross-site scripting attacks is more tricky. But the basic strategy has to be to decide how you will escape user input, decide where you will escape it, and always do it in the same place. Don't fall into the trap of thinking that more is better! Consistently doing it in one way in one place will suffice, and will avoid your having to figure out which of the multiple levels of escaping are causing a specific bug.

Or course learning how to create and maintain a sane architecture takes experience. And moreover, it takes reflecting on your bad experiences. So pay attention to your current pain points (it looks like you are), and think about what you could have done differently to avoid them. If you have a mentor, talk with your mentor. That won't always help you so much with this project, but it will with the next.

Answer 3

To prevent sql injection attacks just do your queries with prepared statements (the exact way will depend on your platform). Once you do that, you'll never have to bother with this particular aspect again. You just need to use this everywhere.

As for general input validation, it's always good to rely on a common base to test for required fields, numbers, etc. ASP.Net's validators are very easy to use for example. One rule of thumb you should follow is not to trust client-side (javascript) to do this for you, since it's easy to go around it. Always do it server-side first.

A special case to keep under your radar is when you allow rich content to be introduced that may contain html/javascript. This can allow a malicious user to inject javascript in your data that will trigger code you don't control when you render it. Do not try to roll your own validation code. Search the web for free, tested, mantained code that will do it for you. Jeff had a few pointers in that regard in one of the podcasts.

Once you automate your input validation code, the time spent doing it should be directly related to the complexity of your business rules. So as a general rule: keep them simple.

Answer 4

@Jeremy - some PHP specifics

When it comes to database queries, always try and use prepared parameterised queries. The mysqli and PDO libraries support this. This is infinitely safer than using escaping functions such as mysql_real_escape_string.

Yes, mysql_real_escape_string is effectively just a string escaping function. It is not a magic bullet. All it will do is escape dangerous characters in order that they can be safe to use in a single query string. However, if you do not sanitise your inputs beforehand, then you will be vulnerable to certain attack vectors.

Imagine the following SQL:

$result = "SELECT fields FROM table WHERE id = ".mysql_real_escape_string($_POST['id']);

You should be able to see that this is vulnerable to exploit.
Imagine the id parameter contained the common attack vector:

1 OR 1=1

There's no risky chars in their to encode, so it will pass straight through the escaping filter. Leaving us:

SELECT fields FROM table WHERE id = 1 OR 1=1

Which is a lovely SQL injection vector.

Whilst these functions are useful, they must be used with care. You need to ensure that all web inputs are validated to some degree. In this case, we see that we can be exploited because we didn't check that a variable we were using as a number, was actually numeric. In PHP you should widely use a set of functions to check that inputs are integers, floats, alphanumeric etc. But when it comes to SQL, heed most the value of the prepared statement. The above code would have been secure if it was a prepared statement as the database functions would have known that 1 OR 1=1 is not a valid literal.

As for htmlspecialchars(). That's a minefield of its own.

There's a real problem in PHP in that it has a whole selection of different html-related escaping functions, and no clear guidance on exactly which functions do what.

Firstly, if you are inside an HTML tag, you are in real trouble. Look at

echo '<img src= "' . htmlspecialchars($_GET['imagesrc']) . '" />';

We're already inside an HTML tag, so we don't need < or > to do anything dangerous. Our attack vector could just be javascript:alert(document.cookie)

Now resultant HTML looks like

<img src= "javascript:alert(document.cookie)" />

The attack gets straight through.

It gets worse. Why? because htmlspecialchars only encodes double quotes and not single. So if we had

echo "<img src= '" . htmlspecialchars($_GET['imagesrc']) . ". />";

Our evil attacker can now inject whole new parameters

pic.png' onclick='location.href=xxx' onmouseover='...

gives us

<img src='pic.png' onclick='location.href=xxx' onmouseover='...' />

In these cases, there is no magic bullet, you just have to santise the input yourself. If you try and filter out bad characters you will surely fail. Take a whitelist approach and only let through the chars which are good. Look at the XSS cheat sheet for examples on how diverse vectors can be

Even if you use htmlspecialchars($string) outside of HTML tags, you are still vulnerable to multi-byte charset attack vectors.

The most effective you can be is to use the a combination of mb_convert_encoding and htmlentities as follows.

$str = mb_convert_encoding($str, ‘UTF-8′, ‘UTF-8′);
$str = htmlentities($str, ENT_QUOTES, ‘UTF-8′);

Even this leaves IE6 vulnerable, because of the way it handles UTF. However, you could fall back to a more limited encoding, such as ISO-8859-1, until IE6 usage drops off.

Answer 5

Just a note on prepared statements. First of all, you should try to use stored procedures if you can... they are probably a better solution in most cases.

Secondly, they both protect you from SQL injection only as long as you don't use dynamic SQL, that is, SQL that writes more SQL and then executes it. In this case they will be ineffective -- and so will be stored procedures.

Regarding the percentage of time you spend: validation is very important and it does take some thought, if not some time. But the percentage depends on how large your application is, no? In a very small application, say, only a newsletter registration, it's quite possible that validation takes a huge percentage of your time.

In larger applications, i.e. where you have a lot of non-presentation code, it is not normal.

Answer 6

I see your problem. It looks like you have the protection logic sprinkled all over the codebase. And each time you write potentially dangerous code you have to be careful to include all protections. And each time a new threat is out, you have to go through all these statements and verify that they're secure.

You can't do real security this way.

You should have some wrappers, that would make producing insecure code hard, if not impossible. For example, prepared statements. But you may want to use an ORM, like Ruby on Rails' ActiveRecord, or some equivalent in your framework.

For output and XSS protection, make sure that output is HTML-escaped by default. Then if you really need to output generated HTML to user, you'll do this explicitly, and it will be easier to verify.

For CSRF protection try to also find a generic solution. Generally it should do its duty automatically, without the need for you to explicitly create a verification token, and manually verify it, discard it, or deny the request.

Answer 7

You're facing a problem that only can be solved by generalizing.

Try to identify common types of input-validation you need

• numeric / string values / regex validation
• range / length
• escaping special characters
• check for common keywords you don't expect in a particular context ('script', 'select', 'drop'...) against a blacklist

and call them systematically before processing the data.

All database access must be done with prepared statements and not concatenating a query string.

All output must be escaped, since you d'ont want to store everything escaped in your database.

A good out of band/social approach is: identify your users the best you can. The higher the chance to get identified, the lesser they will fool with the system. Get their mobile phone number to send a code, check their credit-card etc.