views:

301

answers:

3

Would you guys show me a way to simulating concurrent access to memcache on Google App Engine? I'm trying with LocalServiceTestHelpers and threads but don't have any luck. Every time I try to access Memcache within a thread, then I get this error:

ApiProxy$CallNotFoundException: The API package 'memcache' or call 'Increment()' was not found

I guess that the testing library of GAE SDK tried to mimic the real environment and thus setup the environment for only one thread (the thread that running the test) which cannot be seen by other thread.

Here is a piece of code that can reproduce the problem

package org.seamoo.cache.memcacheImpl;

import org.testng.Assert;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import com.google.appengine.api.memcache.MemcacheService;
import com.google.appengine.api.memcache.MemcacheServiceFactory;
import com.google.appengine.tools.development.testing.LocalMemcacheServiceTestConfig;
import com.google.appengine.tools.development.testing.LocalServiceTestHelper;

public class MemcacheTest {
 LocalServiceTestHelper helper;

 public MemcacheTest() {
  LocalMemcacheServiceTestConfig memcacheConfig = new LocalMemcacheServiceTestConfig();
  helper = new LocalServiceTestHelper(memcacheConfig);
 }

 /**
  * 
  */
 @BeforeMethod
 public void setUp() {
  helper.setUp();
 }

 /**
  * @see LocalServiceTest#tearDown()
  */
 @AfterMethod
 public void tearDown() {
  helper.tearDown();
 }

 @Test
 public void memcacheConcurrentAccess() throws InterruptedException {
  final MemcacheService service = MemcacheServiceFactory.getMemcacheService();
  Runnable runner = new Runnable() {

   @Override
   public void run() {
    // TODO Auto-generated method stub
    service.increment("test-key", 1L, 1L);
    try {
     Thread.sleep(200L);
    } catch (InterruptedException e) {
     // TODO Auto-generated catch block
     e.printStackTrace();
    }
    service.increment("test-key", 1L, 1L);
   }
  };

  Thread t1 = new Thread(runner);
  Thread t2 = new Thread(runner);
  t1.start();
  t2.start();
  while (t1.isAlive()) {
   Thread.sleep(100L);
  }
  Assert.assertEquals((Long) (service.get("test-key")), new Long(4L));
 }
}
A: 

I guess that the testing library of GAE SDK tried to mimic the real environment and thus setup the environment for only one thread (the thread that running the test) which cannot be seen by other thread.

This guess is roughly correct: since GAE itself never runs multiple threads within one process, therefore similarly neither does the SDK -- so it's not clear to me why you want to test a multithreaded access that just can't happen in GAE.

Alex Martelli
"since GAE itself never runs multiple threads within one process". Nitpicking, but there is never more than one request handling thread, and you cannot start your own thread, but there are some things that can happen asynchronously (like all RPC calls), so I assume that internally there are multiple threads.
Thilo
@Thilo, not really: asynchronous operation is not necessarily synonymous with threading (remember that threading is surprisingly limited in Python). One correct nitpick is that I could have limited my assertion to GAE _in Python_ -- the Java runtime is different (and I'm not really familiar with it, but I do believe it allows multithreaded usage).
Alex Martelli
GAE is basically a servlet engine. It has one request per dedicated thread (could be implemented as processes) for the duration of the call.
Romain Hippeau
@Romain, in the case of _Python_ GAE, it's definitely processes, not threads -- in particular, if during a query's service your code sets any global variables, the values you've set will stay put until the end of the query service (no other service to another query is ever allowed to trample on them or even see them). After a query's done, a process may be terminated, or reused for the next query, at the engine's whim -- but other queries being serviced simultaneously to this one are _definitely_ never being served in the same process.
Alex Martelli
@Alex Martelli, All Java Servlet Engines I know of are Thread based. It would be theoretically possible to implement one with Processes.In a Servlet, that is what you use if you use the Java API, There are no global variables possible, There is an Application Context that allows you to put variables in at the Application level (There can be multiple Servlets) and there is a Servlet Context that allows you to put information in at a Servlet Level (There is also a Session context and a request context for session scope and request scope respectively)I suspect the Py and Java impl are different
Romain Hippeau
The App Engine Java runtime is currently single-threaded. This may not be the case in future, however, so relying on this accident of implementation would be a bad idea.
Nick Johnson
+1  A: 

GAE does not support User Threads. I think you are trying to over-complicate.

One way you could test this is by creating an external Threaded client and spawning as many threads as you want to access GAE remotely. If it running on your local box just point there.

Romain Hippeau
+1. The only meaningful way to test is to have multiple external clients hammering the web service simultaneously.
Thilo
I'm dealing with a scenario when my app is running on two different servers and try to modify the same memcache entry (at the same time) and threading is the stuff that I find available to mimic this scenario.
Phương Nguyễn
I really don't want to create external threaded client, since it would request my unit tests to depend on a web server (which, well, request little heavy starting overhead, and cannot be done in a unit-way).
Phương Nguyễn
Hey, there is one way that I can think of, is to forward all call to Memcache in any threads to a given thread (with the aid of mocking). (The re-actor pattern I think). Do you think it feasible?
Phương Nguyễn
@Phuong Nguyen de ManCity fan It sounds like you are trying to test if concurrency works in App Engine ?If these are unit tests all you need to do is test that you can write and read from Memcache, Unit testing is a functional activity.
Romain Hippeau
+3  A: 

Are you trying to test your app, or App Engine's memcache implementation? The semantics of memcache under concurrent reads and writes are well understood - you'd be better off simply simulating the conditions that can occur in order to verify your app handles them fine.

Nick Johnson
Hmm, I think you made a good point there. I don't really need to test concurrent access since it should be guaranteed by Google. What I need to test is whether my business logic work well under synchronized access.
Phương Nguyễn