retry connection to rate limited URL until successful in Scala

Question

I need make a GET call to a REST api which is rate limited. I can find out what the current rate limit is by making a call and checking the HTTP headers. If I've exceeded my rate limit, I should wait for a bit before retrying. I'd like to write something like:

val conn = connect(url, _.getHeaderField("X-RateLimit-Remaining").toInt > 0, 500)

I have a working solution using a var, a while loop and some repetitious code, but it feels clunky:

def connect(url: String, succeeded: URLConnection=>Boolean, waitMillis: Int) = {
  var conn = new URL(url).openConnection
  while (!succeeded(conn)) {
    Thread.sleep(waitMillis)
    conn = new URL(url).openConnection
  }
  conn
}

Is there a cleaner way to do this?

Answer 1

You could make it tail-recursive:

def connect(url: String, succeeded: URLConnection=>Boolean, wait: Int): URLConnection = {
  val conn = new URL(url).openConnection
  if (succeeded(conn)) conn
  else {
    Thread.sleep(wait)
    connect(url,succeeded,wait)
  }
}

Or you could use an infinite iterator pattern, either raw:

def connect(url: String, succeeded: URLConnection=>Boolean, waitMillis: Int) = {
  val tries = Iterator.continually( new URL(url).openConnection )
  tries.dropWhile(
    conn => if (succeeded(conn)) false else { Thread.sleep(waitMillis); true }
  ).next
}

or by wrapping the URL call in a wait that returns an option (especially useful if you want option handling elsewhere; not sure whether you want to embed the wait there or outside):

def attemptConnect(url: String, succeeded: URLConnection=>Boolean, waitMillis: Int) = {
  val conn = new URL(url).openConnection
  if (succeeded(conn)) Some(conn)
  else { Thread.sleep(waitMillis); None }
}
def connect(url: String, succeeded: URLConnection=>Boolean, waitMillis: Int) = {
  val tries = Iterator.continually( attemptConnect(url,succeeded,waitMillis) )
  tries.dropWhile(_.isEmpty).next
}