Parsing json with sed and awk

Question

I'm trying to parse json returned from a curl request, like sp:

curl 'http://twitter.com/users/username.json' | sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}'

I have it set working where it splits the json into fields, i.e. the above returns

% ...
"geo_enabled":false
"friends_count":245
"profile_text_color":"000000"
"status":"in_reply_to_screen_name":null
"source":"web"
"truncated":false
"text":"My status"
"favorited":false
% ...

But what I would like to do is grab a specific field (denoted by the -v k=text) and only print that.

Any ideas?

Answer 1

I've never used it, but you could try out jsawk. It would be something like this (haven't tested this, so I may be wrong):

curl 'http://twitter.com/users/username.json' | jsawk -a 'return this.name'

Answer 2

please don't do it!

do not use line-oriented tools to parse hierarchical data serialized into text. it works only for special cases and will haunt you and other people. if you really can't use a ready-made json parser, write a simple one using recursive descent. it's easy and will endure changes the emitting side justly considers cosmetic (added or removed whitespace including newlines).

Answer 3

Use Python's JSON support instead of using awk!

See http://docs.python.org/library/json.html

EDIT

Added explicit plea to use Python JSON support in place of awk.

Answer 4

You've asked how to shoot yourself in the foot and I'm here to provide the ammo:

curl -s 'http://twitter.com/users/username.json' | sed -e 's/[{}]/''/g' | awk -v RS=',"' -F: '/^text/ {print $2}'

You could use tr -d '{}' instead of sed. But leaving them out completely seems to have the desired effect as well.

If you want to strip off the outer quotes, pipe the result of the above through sed 's/\(^"\|"$\)//g'

I think others have sounded sufficient alarm. I'll be standing by with a cell phone to call an ambulance. Fire when ready.

Answer 5

here's one way you can do it with awk

curl -sL 'http://twitter.com/users/username.json' | awk -F"," -v k="text" '{
    gsub(/{|}/,"")
    for(i=1;i<=NF;i++){
        if ( $i ~ k ){
            print $i
        }
    }
}'

Answer 6

How about using Rhino? It's a command-line JavaScript tool. Unfortunately, it's a bit rough for this type of application. It doesn't read from stdin very well.