basic json > struct question ( using 'Go')

Question

I'm working with twitter's api, trying to get the json data from

http://search.twitter.com/trends/current.json

which looks like:

{"as_of":1268069036,"trends":{"2010-03-08 17:23:56":[{"name":"Happy Women's Day","query":"\"Happy Women's Day\" OR \"Women's Day\""},{"name":"#MusicMonday","query":"#MusicMonday"},{"name":"#MM","query":"#MM"},{"name":"Oscars","query":"Oscars OR #oscars"},{"name":"#nooffense","query":"#nooffense"},{"name":"Hurt Locker","query":"\"Hurt Locker\""},{"name":"Justin Bieber","query":"\"Justin Bieber\""},{"name":"Cmon","query":"Cmon"},{"name":"My World 2","query":"\"My World 2\""},{"name":"Sandra Bullock","query":"\"Sandra Bullock\""}]}}

My structs look like:

type trend struct {  
 name  string  
 query string  
}  

type trends struct {  
 id string  
 arr_of_trends []trend  
}  

type Trending struct {  
 as_of  string  
 trends_obj trends  
}

and then I parse the JSON into a variable of type Trending. I'm very new to JSON so my main concern is making sure I've have the data structure correctly setup to hold the returned json data.

I'm writing this in 'Go' for a project for school. (This is not part of a particular assignment, just something I'm demo-ing for a presentation on the language)

UPDATE: In accordance with PeterSO's comment I'm going the regexp route. Using:

Cur_Trends := new(Current)
/* unmarshal the JSON into our structures */

//find proper json time-name
aoUnixTime, _, _ := os.Time()

// insert code to find and convert as_of Unix time to aoUnixTime
    aoName := time.SecondsToUTC(aoUnixTime).Format(`"2006-01-02"`)
    fmt.Printf("%s\n", aoName)
    regexp_pattern := "/" + aoName + "/"
    regex, _ := regexp.Compile(regexp_pattern);

    cleaned_json := regex.ReplaceAllString(string(body2), "ntrends")
    os.Stdout.WriteString(cleaned_json)

Doesn't show any changes. Am I specifying the regexp wrong? It seems like 'Go' only allows for one regexp at a time...

UPDATE: can now change the date/time to "ntrends" but the "Unmarshaling" isn't working. I can get everything moved into an interface using json.Decode, but can't iterate through it...

I guess my new question is, How do I iterate through:

map[as_of:1.268176902e+09 trends:map[ntrends:[map[name:#nowplaying query:#nowplaying] map[name:#imtiredofseeing query:#imtiredofseeing] map[name:#iWillNever query:#iWillNever] map[name:#inmyfamily query:#inmyfamily] map[name:#raiseyourhandif query:#raiseyourhandif] map[name:#ripbig query:#ripbig] map[name:QVC query:QVC] map[name:#nooffense query:#nooffense] map[name:#RIPLaylaGrace query:#RIPLaylaGrace] map[name:Justin Bieber query:"Justin Bieber"]]]]

using "for...range" is giving me weird stuff...

Answer 1

Ugh, this seems like JSON that Go can't parse. Twitter pulls this kind of weird stuff all the time in their API.

The 'trends' object is a map from date objects to an array of trending topics. Unfortunately the Go JSON parser isn't smart enough to handle this.

In the meantime you could manually parse this format, or just do a regular expression search for the topics.

Either way, I'd post this as a Go issue and see what they say: http://code.google.com/p/go/issues/list

Answer 2

Twitter is famous for its Fail Whale, and the Twitter API gets a failing grade too; it's horrible.

Twitter trends current Search API method response can be expressed (using just two trends to simplify the examples) in canonical, normalized JSON form as:

{
    "as_of":1268069036,
    "trends":[
        {"name":"Happy Women's Day","query":"\"Happy Women's Day\" OR \"Women's Day\""},
        {"name":"#MusicMonday","query":"#MusicMonday"},{"name":"#MM","query":"#MM"}
    ]
}

The as_of date is in Unix time, the number of seconds since 1/1/1970.

In Go, this can be described by:

type Trend struct {
    Name  string
    Query string
}

type Current struct {
    As_of  int64
    Trends []Trend
}

Twitter mangles the canonical, normalized JSON form to become:

{
    "as_of":1268069036,
    "trends":{
        "2010-03-08 17:23:56":[
            {"name":"Happy Women's Day","query":"\"Happy Women's Day\" OR \"Women's Day\""},
            {"name":"#MusicMonday","query":"#MusicMonday"}
        ]
    }
}

Sometimes, Twitter returns this equivalent JSON form.

{
    "trends":{
        "2010-03-08 17:23:56":[
            {"name":"Happy Women's Day","query":"\"Happy Women's Day\" OR \"Women's Day\""},
            {"name":"#MusicMonday","query":"#MusicMonday"}
        ]
    },
    "as_of":1268069036
}

"2010-03-08 17:23:56": is a JSON object name. However, it's -- nonsensically -- a string form of as_of.

If we replace "2010-03-08 17:23:56": by the object name "ntrends": (for nested trends), overwriting the redundant as_of string time, we have the following revised Twitter JSON form:

{
    "as_of":1268069036,
    "trends":{
        "ntrends":[
            {"name":"Happy Women's Day","query":"\"Happy Women's Day\" OR \"Women's Day\""},
            {"name":"#MusicMonday","query":"#MusicMonday"}
        ]
    }
}

It's easy to scan the Twitter JSON form for "as_of":, read the following number as the as_of Unix time, and convert it to JSON name form e.g.:

var aoUnixTime int64
// insert code to find and convert as_of Unix time to aoUnixTime
aoName := time.SecondsToUTC(aoUnix).Format(`"2006-01-02 15:04:05":`)

Now we can scan the Twitter JSON form for the aoName value and replace it with "ntrends": to get the revised Twitter JSON form.

In Go, the revised Twitter JSON form can be described by:

type Trend struct {
    Name  string
    Query string
}

type NTrends struct {
    NTrends []Trend
}

type Current struct {
    As_of  int64
    Trends NTrends
}

Note: the first character of the struct and field identifiers are uppercase so that they can be exported.

I've programmed and tested this approach and it seems to work. Since this is a school project for you, I haven't published my code.

Answer 3

Revision to earlier answer.

The Twitter Search API Method trends response is in convenient canonical and normalized JSON form:

{"trends":[{"name":"#amazonfail","url":"http:\/\/search.twitter.com\/search?q=%23amazonfail"},... truncated ...],"as_of":"Mon, 13 Apr 2009 20:48:29 +0000"}

The Twitter Search API Methods trends current, daily and weekly responses are, however, in an unnecessarily inconvenient JSON form similar to:

{"trends":{"2009-03-19 21:00":[{"query":"AIG","name":"AIG"},... truncated ...],... truncated ...},"as_of":1239656409}

In clear violation of the rules for the encapsulation of algorithms and data structures, this unnecessarily discloses that currently these methods use a map or dictionary for their implementation.

To read the JSON data from Twitter current trends into Go data structures, using the json package, we can do something similar to the following.

package main

import (
    "fmt"
    "json"
)

type Trend struct {
    Name  string
    Query string
}

type Current struct {
    As_of  int64
    Trends map[string][]Trend
}

var currentTrends = `{"as_of":1268069036,"trends":{"2010-03-08 17:23:56":[{"name":"Happy Women's Day","query":"\"Happy Women's Day\" OR \"Women's Day\""},{"name":"#MusicMonday","query":"#MusicMonday"},{"name":"#MM","query":"#MM"},{"name":"Oscars","query":"Oscars OR #oscars"},{"name":"#nooffense","query":"#nooffense"},{"name":"Hurt Locker","query":"\"Hurt Locker\""},{"name":"Justin Bieber","query":"\"Justin Bieber\""},{"name":"Cmon","query":"Cmon"},{"name":"My World 2","query":"\"My World 2\""},{"name":"Sandra Bullock","query":"\"Sandra Bullock\""}]}}`

func main() {
    var ctJson = currentTrends
    var ctVal = Current{}
    ok, errtok := json.Unmarshal(ctJson, &ctVal)
    if !ok {
        fmt.Println("Unmarshal errtok: ", errtok)
    }
    fmt.Println(ctVal)
}