What ist a RESTful-resource in the context of large data sets, i.E. weather data?

Question

So I am working on a webservice to access our weather forecast data (10000 locations, 40 parameters each, hourly values for the next 14 days = about 130 million values).

So I read all about RESTful services and its ideology.

So I understand that an URL is adressing a ressource.

But what is a ressource in my case?

The common use case is that you want to get the data for a couple of parameters over a timespan at one or more location. So clearly giving every value its own URL is not pratical and would result in hundreds of requests. I have the feeling that my specific problem doesn't excactly fit into the RESTful pattern.

Update: To clarify: There are two usage patterns of the service. 1. Raw data; rows and rows of data for several locations and parameters.

1. Interpreted data; the raw data calculated into symbols (Suns & clouds, for example) and other parameters.

There is not one 'forecast'. Different clients have different needs for data.

The reason I think this doesn't fit into the REST-pattern is, that while I can actually have a 'forecast' ressource, I still have to submit a lot of request parameters. So a simple GET-request on a ressource doesn't work, I end up POSTing data all over the place.

Answer 1

Maybe you can use forecast as the ressource and go deeper to fine grained services with xlink.

Answer 2

Would it be possible to do something like this,Since you have so many parameters so i was thinking if somehow you can relate it to a mix of id / parameter combo to decrease the url size

/WeatherForeCastService//day/hour

Answer 3

www.weatherornot.com/today/days/x       // (where x is number of days)
www.weatherornot.com/today/9am/hours/h  // (where h is number of hours)

Answer 4

Firstly, there is no once-and-for-all right answer.

Each valid url is something that makes sense to query, think of them as equivalents to providing query forms for people looking for your data - that might help you narrow down the scenarios.

It is a matter of personal taste and possibly the toolkit you use, as to what goes into the basic url path and what parameters are encoded. The debate is a bit like the XML debate over putting values in elements vs attributes. It is not always a rational or logically decided issue nor will everybody be kind in their comments on your decisions.

If you are using a backend like Rails, that implies certain conventions. Even if you're not using Rails, it makes sense to work in the same way unless you have a strong reason to change. That way, people writing clients to talk to Rails-based services will find yours easier to understand and it saves you on documentation time ;-)

Answer 5

So I am working on a webservice to access our weather forecast data (10000 locations, 40 parameters each, hourly values for the next 14 days = about 130 million values). ... But what is a ressource in my case?

That depends on the details of your problem domain. Simply having a large amount of data is not a good reason to avoid REST. There are smart ways and dumb ways to model and expose that data.

As you rightly see, your main goal at this point should be to understand what exactly a resource is. Knowing only enough about weather forecasting to follow the Weather Channel, I won't be much help here. It's for domain experts like yourself to make that call.

If you were to explain in a little more detail the major domain concepts you're working with, it might make it a little easier to give specific advice.

For example, one resource might be Forecast. When weatherpeople talk about Forecasts, what words keep coming up? When you think about breaking a forecast down into smaller elements, what words do you use to describe the pieces?

Do this process recursively, and you'll probably be able to make a list of important terms. Don't forget that these terms can describe things or actions. Think about what these terms really mean, what data you can use to model them, how they can be aggregated.

At this point you'll have the makings of something you can start building a RESTful system around - but not before.

Don't forget that a RESTful system is not a data dump wrapped in HTTP - it's a hypertext-driven system.

Also don't forget that media types are the point of contact between your server and its clients. A media type is only limited by your imagination and can model datasets of any size if you're clever about it. It can contain XML, JSON, YAML, binary elements such as a Bloom Filter, or whatever works for the problem.