regex that matches a string that contains some text

Question

I need a regex that matches

re.compile('userpage')


href="www.example.com?u=userpage&as=233&p=1"
href="www.example.com?u=userpage&as=233&p=2"

I want to get all urls that have u=userpage and p=1

How can I modify the regex above to find both u=userpage and p=1?

Answer 1

/((u=userpage).*?(p=1))|((p=1).*?(u=userpage))/

This will get all strings that contain the two bits you're looking for.

Answer 2

To make sure you don't accidentally match parts like bu=userpage, u=userpagezap, p=111 or zap=1, you need abundant use of the \b "word-boundary" RE pattern element. I.e.:

re.compile(r'\bp=1\b.*\bu=userpage\b|\bu=userpage\b.*\bp=1\b')

The word-boundary elements in the RE's pattern prevent the above-mentioned, presumably-undesirable "accidental" matches. Of course, if in your application they're not "undesirable", i.e., if you positively want to match p=123 and the like, you can easily remove some or all of the word-boundary elements above!-)

Answer 3

import lxml.html, urlparse

d = lxml.html.parse(...)
for link in d.xpath('//a/@href'):
    url = urlparse.urlparse(link)
    if not url.query:
        continue
    params = urlparse.parse_qs(url.query)
    if 'userpage' in params.get('u', []) and '1' in params.get('p', []):
        print link

Answer 4

It is possible to do this with string hacking, but you shouldn't. It's already in the standard library:

>>> import urllib.parse
>>> urllib.parse.parse_qs("u=userpage&as=233&p=1")
{'u': ['userpage'], 'as': ['233'], 'p': ['1']}

and hence

import urllib.parse
def filtered_urls( urls ):
    for url in urls:
        try:
            attrs = urllib.parse.parse_qs( url.split( "?" )[ 1 ] )
        except IndexError:
            continue

        if "userpage" in attrs.get( "u", "" ) and "1" in attrs.get( "p", "" ):
            yield url

foo = [ "www.example.com?u=userpage&as=233&p=1", "www.example.com?u=userpage&as=233&p=2" ]

print( list( filtered_urls( foo ) ) )

Note that this is Python 3 -- in Python parse_qs is in urlparse instead.

Answer 5

if you want to use, in my opinion, something more proper approach, than regexp:

from urlparse import *
urlparsed = urlparse('www.example.com?u=userpage&as=233&p=1')
# -> ParseResult(scheme='', netloc='', path='www.example.com', params='', query='u=userpage&as=233&p=1', fragment='')
qdict = dict(parse_qsl(urlparsed.query))
# -> {'as': '233', 'p': '1', 'u': 'userpage'}
qdict.get('p') == '1' and qdict.get('u') == 'userpage'
# -> True

Answer 6

Regex is not a good choice for this because 1) the params could appear in either order, and 2) you need to do extra checks for query separators so that you don't match potential oddities like "flu=userpage", "sp=1", "u=userpage%20haha", or "s=123". (Note: I missed two of those cases in my first pass! So did others.) Also: 3) you already have a good URL parsing library in Python which does the work for you.

With regex you'd need something clumsy like:

q = re.compile(r'([?&]u=userpage&(.*&)?p=1(&|$))|([?&]p=1&(.*&)?u=userpage(&|$))')
return q.search(href) is not None

With urlparse you can do this. urlparse gives you a little more than you want but you can use a helper function to keep the result simple:

def has_qparam(qs, key, value):
    return value in qs.get(key, [])

qs = urlparse.parse_qs(urlparse.urlparse(href).query)
return has_qparam(qs, 'u', 'userpage') and has_qparam(qs, 'p', '1')