We have a need to poll a number of external data sources.
The data sources initially are SOAP web services, but they might be in future RESTful web services, flat files, or other things.
The polling period needs to be configurable (ideally per-data-source).
The returned data initially is XML, but in future it might a line from a file or something else.
The returned value needs to be filtered to see if it's relevant. This might be it contains a particular string or regexp, or for XML maybe we might want to test for a specific node, text value (or full XPath expression).
It would be nice to be able to transform the returned value (do a regexp replace, or XSLT transform)
The original or transformed response then needs to be stored somewhere searchable (so probably in an Oracle DB, but could be elsewhere)
We can knock this up in many ways - most languages have readily-available libraries to to the various bits, but there's enough complexity here that I'm posting this to ask if anyone is aware of a framework or tool that we could use for this?
It's close to a monitoring solution except that we're interested in the retreived values, not just whether the service is up and how fast it's responding.