ansaurus

Question

Getting a count of returns seen by a RESTful request

Answer 1

A:

Why not have the REST webservice just return the data as a JSON or XML, and in there you can have a property about length.

James Black 2009-10-23 01:13:32

I don't want to get all the results at once. I'm trying to populate a table in batches, but I want to know what the total count will be for the full result set.

georryan 2009-10-26 19:13:35

When you make the initial request you get back the first batch and a total number, or, a number that is still remaining. Then you just loop through and get the next batch, until you have gotten all the information. But, either the server-side is stateful, or you are making extra queries to the database to get a limited number of results each time. And what happens if you are returning it back sorted, and you are on G, but something that starts with C is inserted into the database, you won't be able to show the new change. Or, just sort by ID.

James Black 2009-10-26 19:26:44

You are saying that the xml or json response will have a length embedded in it already? Where is that? Is it in the header?I'm actually using .net to create my RESTful output, and I don't see any count on the xml output.

georryan 2009-10-26 19:41:19

You would pass back a property ('numberremaining') so that every time you pass data back, this number should be decreasing, but you would just put it in, then put in the data you want to pass back. <mydata><numremain>500</numremain><data>...</data></mydata>

James Black 2009-10-26 19:44:56

Answer 2

+3 A:

Other people might have objections to this concept, but, this seems reasonable to me:

HEAD /your/api HTTP/1.1

HTTP/1.1 200 OK
Date: Fri, 23 Oct 2009 00:58:17 GMT
Content-Type: application/xml; charset=UTF-8
Content-Length: 89
X-Result-Count: 100000000

And then:

GET /your/api HTTP/1.1

HTTP/1.1 200 OK
Date: Fri, 23 Oct 2009 00:58:17 GMT
Content-Type: application/xml; charset=UTF-8
Content-Length: 89
X-Result-Count: 100000000

<?xml version="1.0" encoding="UTF-8"?>
<results>
  100000000 results go here.
</results>

Note: A HEAD request is used here to obtain the count without having to pull the full data set. HEAD requests retrieve only the HTTP headers, not the body of the response.

This would be the most RESTful way I can think of indicating how many results you're gonna get back before you send it over the wire. The main trick is just coming up with the best header name for it. X-Result-Count is decent, but if you can find prior art and reuse their header name choice, that would be even better (as long as they didn't name it something really dumb). That said, I don't expect you'll have much luck, so you should probably stick with X-Result-Count.

Also, I think you may have misunderstood what "REST" actually entails. There's no reason you can't give a representation by range. For example:

GET /your/api?page=1&perpage=10 HTTP/1.1

HTTP/1.1 200 OK
Date: Fri, 23 Oct 2009 00:58:17 GMT
Content-Type: application/xml; charset=UTF-8
Content-Length: 101
X-Result-Count: 10

<?xml version="1.0" encoding="UTF-8"?>
<results>
  First 10 results of 100000000 go here.
</results>

However, to be RESTful, you need to be able to tell the client about the representation identified by /your/api?range=0-9 or /your/api?page=1&perpage=10 without using out-of-band information. For example, if your /your/api page would return too many results, do a temporary redirect to /your/api?page=1&perpage=10, and include hyperlinks to /your/api?page=2&perpage=10. Note that a hyperlink in this context could be something simple like:

<?xml version="1.0" encoding="UTF-8"?>
<results>
  <result>
    This is a result.
  </result>
  <result>
    This is also a result.
  </result>
  <link rel="next" href="/your/api?page=3&perpage=2" />
  <link rel="prev" href="/your/api?page=1&perpage=2" />
</results>

Now the information to navigate the results of your API calls is in-band and actually RESTful.

Essentially, REST is plain-old-HTTP with caching done right and usually sensible URIs thrown in for good measure. It's also "hypertext as the engine of application state" (i.e. resources should link to other resources). It is not a protocol, it's an architectural style. Anyone who tells you differently had better be named Roy Fielding.

Addenda:

If you want to indicate the total count versus the page count, you can define the header like so:

X-Result-Count: 0-9/100000000

Or adjust as necessary.

Bob Aman 2009-10-23 01:27:38

Hmm, it looks like your count is lumped into the full set of results that come with that count, is that correct? So by that I mean that if I'm selecting just the top 10 out of a full set, the count would say 10, and I'd get the 10 items along with that.What I'm shooting for is a way of getting the count for a request of all the items, even though I want to grab small sets of the full data using (page=, top=, next=, etc).I'd like the get the count so I can get an accurate number for the size of rows needed for the table the information will be fitting into.Any ideas?

georryan 2009-10-26 19:27:43

Also, I don't really want to grab all the data just to get a count of how many there are. That would defeat the purpose of why I need the count.

georryan 2009-10-26 19:29:45

@georryan Right, that's why it's a HEAD request. HEAD request means you only obtain the headers, which in this case would include the count, but not the data. Also, because you're defining the header, you can return whatever data you want.

Bob Aman 2009-10-26 22:03:08

I updated the answer to address some of this.

Bob Aman 2009-10-26 22:06:18

I'm using ADO.Net Data Services to create my REST end point. As it turns out, there is a new version of the framework that gives me the exact information that I need. I can get the the count I'm looking for by using "$inlinecount=allpages" or "$count". I had to download the new version, but it gave me exactly what I needed. All your suggestions definitely got me looking in the direction that helped me find the answer.Thanks!http://mstecharchitect.blogspot.com/2009/09/adonet-data-services-server-driven.htmlhttp://mstecharchitect.blogspot.com/2009/09/adonet-data-services-projections.html

georryan 2009-10-26 22:14:16

Interesting. Could you perhaps add your full solution to this question as an answer? It looks to me like it returns a data feed with the total count embedded in the XML, so you're still getting the first page then?

Bob Aman 2009-10-26 22:19:28

Answer 3

A:

You should be able to take care of this in your REST resource name design. You'd start with something like:

/widget/12345 (the representation of widget 12345)
/widgets (the list of all widget resource names, ie links)

You might quickly decide that "/widgets" will be a humongous list and decide to support pages, something like

/widgets/page/43 (this could have links to the 4200th to 4299th widgets, and additional information like the total number of pages or a count of the widgets.)

In other cases you can subdivide a large set into a natural hierarchy:

/widgets/mahogany, /widgets/oak, ...
/movies/drama, /movies/romance, ...
/computers/harddrives/seagate, /computers/usbdrives/kingston

And you can define queries too:

/widgets?maxprice=200&maxweight=4

Jim Ferrans 2009-10-23 05:55:54

Answer 4

A:

Why don't you make your resource handle queries for that type of metadata? Suppose that

GET /items

returns your list of items like this:

<items count="5" modified="2009-10-22">
  <item url="/items/first" name="First Item" />
  <item url="/items/second" name="Second Item" />
  ...
</items>

Then something like:

GET /items?info

could return an empty list like this:

<items count="5" modified="2009-10-22" type="info" />

or possibly a generic info document like this:

<info>
  <items count="5" modified="2009-10-22" url="/items" />
</info>

You could also implement an "info" resource like this:

GET /info?items&users

which might return:

<info>
  <items count="5" modified="2009-10-22" url="/items" />
  <users count="8" modified="2009-10-05" url="/users" />
</info>

Don McCaughey 2009-10-23 06:18:41

-1 This concept doesn't properly take HTTP caching into account. Combining two resources into a single HTTP request/response reduces your chances for a cache hit. It also prevents the client from using `If-Modified-Since` and `Last-Modified` effectively. Also, `Last-Modified` is the appropriate way of returning that information to the client, and it's perfectly reasonable to do this with a HEAD request. If you're opening a single HTTP session and resources are thousands (or more) of bytes rather than hundreds, there's very little advantage to batching because the session itself is a batch.

Bob Aman 2009-10-23 17:25:56

I'm not quite sure what your saying here -- are you saying that using "GET /items" and "GET /items?info" screws up caching? If so, then instead of the "?info" query, would "GET /items/info" make more sense?

Don McCaughey 2009-10-23 18:47:59

No, I'm saying that returning information about both items and users in the same response is bad. Keep them in separate requests and responses.

Bob Aman 2009-10-23 21:52:14

And also, whenever possible, use HTTP headers for metadata rather than transmit metadata in a separate resource. The only time you should ever consider using a separate resource for metadata is if you have both a ton of it and you need the metadata to actually process the real data itself. i.e., in cases where you absolutely must have the metadata first. The overwhelming majority of web services don't really have much metadata that isn't covered by one of the many standardized HTTP headers, and most of what they do have can go in-line with the data itself.

Bob Aman 2009-10-23 21:55:55

@Bob Aman - Except, the number of items in the collection is _not_ a metadata, it's a genuine data member of the collection resource itself. And passing parts of the resource data as an HTTP header is definitely not a good idea.

Franci Penov 2010-09-16 10:17:47

@Franci Penov — I agree, passing resource data in headers would be a bad idea. However, I don't agree that the total count is strictly not metadata.

Bob Aman 2010-09-16 16:09:02

@Bob Aman - If you look at the Wikipedia definition of metadata, specifically at the metadata types (http://en.wikipedia.org/wiki/Metadata#Metadata_types) you will notice that the items count does not fall into any of these buckets.

Franci Penov 2010-09-16 18:23:46

@Franci Wow, really? Because it's not one of the categories that Wikipedia defines it's not metadata? From the same Wikipedia article: "Metadata is loosely defined as data about data." The count is data about how many data items are in the collection. Hence, metadata. Regardless, this is a pretty stupid argument.

Bob Aman 2010-09-20 08:36:15

ansaurus

tags:

views:

answers:

Getting a count of returns seen by a RESTful request

related questions