Getting data from remote service if not cached in database - suggestion needed

Question

In one of my current applications, I need to get customer data from a remote service (CRM) via Webservice / SOAP. But I also want to cache the data in the mysql database, so that I dont need to connect to the webservice for the next time (data does not change often, remote service is slow and has bandwidth limit - so caching is OK / a must).

I am quite sure about the technical part of this task, but I am not so sure about how to implement this clean and transparent in my web app.

All my other data comes from a single mysql database, so I repositories which return lists or single entities queried from the database with NHibernate.

My ideas so far:

---

1 all in one

Use a CustomerRepository, which looks for the customer by Id, if successfull, then return it, else call the webservice and save the retrieved data to the database.

Controller looks like this:

class Controller
{
    private CustomerRepository Rep;

    public ActionResult SomeAction(int id)
    {
        return Json(Rep.GetCustomerById(id));
    }
}

Repository in pseudo / simple code like this:

class CustomerRepository 
{
    public Customer GetCustomerById(int id)
    {
        var cached = Database.FindByPK(id);
        if(cached != null) return cached;

        var webserviceData = Webservice.GetData(id);
        var customer = ConvertDataToCustomer(webserviceData);

        SaveCustomer(customer);

        return customer;
    }
}

Although the above looks somewhat straightforward, I think the CustomerRepository class will grow quite large and ugly. So I dont like that approach at all.

A repository should only load data from the database, that should be its "contract" in my app at least.

---

2 sepereate and sticked together in the controller

Use seperate classes for the repository (db access) and webservice (remote access) and let the controller do the work:

Controller looks like this:

class Controller
{
    private CustomerRepository Rep;
    private Webservice Service;

    public ActionResult SomeAction(int id)
    {
        var customer = Rep.GetCustomerById(id);
        if(customer != null) return Json(customer);

        var remote = Service.GetCustomerById(id);
        Rep.SaveCustomer(remote);

        return Json(remote);
    }
}

Although this looks a bit better, I still dont like to put all that logic in the controller, because error handling if the service does not return data is omitted and could probably clutter things a bit more.

Maybe I could make another service layer used by the Controller, but the code would be quite the same, but in another class.

Actually I would like to have my Controller use a single Interface/Class, which encapsulates that stuff, but I dont want one class which "does it all": accessing the repository, accessing the webservice, saving the data... it feels somewhat wrong to me...

---

All ideas so far are / will probably become quite bloated with caching code, error handling, etc. I think.

Maybe I could clean up a few things using AOP?

How would you implement stuff like the above?

Technical frameworks used: ASP.NET MVC, Spring.NET for DI, NHibernate as ORM, mysql as database, the remote service is accessed via SOAP.

Answer 1

This is a perfect candidate for a service layer, in my opinion.

You have two data sources (DB Repo and Web Service), and you need to encapsulate those for the controller, as you suggested.

If the service layer has a method such that your controller can call, like this:

public class Controller
{
    private CustomerService _customerService; // Perhaps injected by an IoC container?

    public ActionResult SomeAction(int id)
    {
        return Json(_customerService.GetCustomerById(id));
    }
}

Then the service layer can handle the business logic / caching like so (you could inject the repository):

public class CustomerService 
{
    private readonly ICustomerRepository _customerRepository;

    public CustomerService(ICustomerRepository customerRepository)
    {
        _customerRepository = customerRepository;
    }

    public Customer GetCustomerById(int id)
    {
        var cached = _customerRepository.FindByPK(id);
        if(cached != null) return cached;

        var webserviceData = Webservice.GetData(id);  // You could inject the web service as well...
        var customer = ConvertDataToCustomer(webserviceData);

        _customerRepository.SaveCustomer(customer);

        return customer;
    }
}

Then, you could keep the repository and web service logic in separate classes, or (better yet) separate assemblies.

Hope that helps!

Answer 2

You can implement your architecture cleanly with the Decorator pattern.

Let us imagine that you have an interface called ICustomerRepository.

You first implement ICustomerRepository (let's call it WsCustomerRepository) based on the web service, and don't worry about the database at all in this implementation.

You then implement another ICustomerRepository (MySqlRepository) that only talks to your database.

Finally, you create a third ICustomerRepository (CachingRepository) that implements the caching logic you describe. It starts by querying the 'local' Repository, but if it doesn't get a result from that, it queries the 'remote' Repository and inserts the result in the 'local' Repository before it returns the result.

Your CachingRepository could look like this:

public class CachingRepository : ICustomerRepository
{
    private readonly ICustomerRepository remoteRep;
    private readonly ICustomerRepository localRep;

    public CachingRepository(ICustomerRepository remoteRep, ICustomerRepository localRep)
    {
        this.remoteRep = remoteRep;
        this.localRep = localRep;
    }

    // implement ICustomerRepository...
}

As you can see, the CachingRepository doesn't even have to know about the concrete web service and database, but can concentrate on its Single Responsibility: Caching

This gives you a clean separation of responsibility, and as far as your Controllers are concerned, they only talk to an ICustomerRepository instance.

Answer 3

Personally in this scenario I would reconsider your design. I would think about having a seperate process (perhaps a windows service) that updates your MySql DB from the remote service.

That way clients can always get the data from your local DB without ever having to block for a significant amount of time. If necessary you can still use caching when getting the data out of the MySql DB.

I guess whether this approach is feasible depends on how many customers you have. The approach worked well for me a few years back when getting exchange rates from a Web service that was not particularly reliable.

Answer 4

In general I like Mark's answer, but I would add two important notes.

1. Should you even be caching this in your DB? To me this is the perfect use of application cache. I typically have a caching interface that encapsulates HTTP Application cache and has easy to use methods that automatically check for existence of a key else they call your method and put in cache.. Also you can put an expiration on it so it will auto expire after a certain amount of time.

   var result = CacheManager.Add(() => YourMethodCall(), "CachedStuffKey", new Timespan(0,1,0));

I know this seems complicated but storing it in cache will be faster in terms of performance and gets rid of having to create a new table.

1. I do agree that this type of logic really doesn't belong in a controller, your controller should be pretty stupid and be mostly glue between your UI and your lower layers. Although, this is depends on what type of app you are building. Not all apps need to be 3 tiered super structures.