@Before
public void setUp() throws Exception {
ApiProxy.setEnvironmentForCurrentThread(new TestEnvironment());
ApiProxy.setDelegate(new ApiProxyLocalImpl(new File(".")){});
ApiProxyLocalImpl proxy = (ApiProxyLocalImpl) ApiProxy.getDelegate();
proxy.setProperty(LocalDatastoreService.NO_STORAGE_PROPERTY, Boolean.TRUE.toString());
}

@After
public void tearDown() throws Exception {
ApiProxyLocalImpl proxy = (ApiProxyLocalImpl) ApiProxy.getDelegate();
LocalDatastoreService datastoreService = (LocalDatastoreService) proxy.getService("datastore_v3");
datastoreService.clearProfiles();
ApiProxy.setDelegate(null);
ApiProxy.setEnvironmentForCurrentThread(null);
}

You discuss an area in your post that I am stuck on. I am fluent in web app development in Java. And I am evaluating GAE for my strt-up too. My choice for sticking with GAE is only its giving me managed hosting for some popular Java web frameworks (I will use Struts). I dont want to mess with a naked server and putty it all the day in solving configuration issues in my web app. So this my motivation behind thinking about GAE.

So far I have code some servlets in it to understand its environment. But most concerned are for me is DB. I am unable to find any Java code snippet about interacting with Database. They are not supporting relational database – that increases my concern about data. How I will see my data or as you said how I will run queries on it.

I will love to hear if you point me towards some better understanding of DB stuff. Is there any tutorial or some code snippet?

Comments are welcomed.

Even Java isn’t what I’m looking for. What I’d like is a maximum of 100MB of easily-persistable objects that can be programmed very flexibly and (because they are all in memory) very efficiently. Maybe JavaScript/V8 on the server and/or an in-memory version of Map/Reduce. I’m afraid I’m not terribly articulate about this, because it’s just a feeling at this point. I’ve been through this enough times to know that I may be just beginning to uncover something very interesting or not.

AppEngine does a brilliant job of its location on the scaling/flexibility tradeoff, I’m just looking for different spot.

With these limitations in mind, you need to do some development in order to get at your data reliably.

First, since you’re pulling logs, I assume you’re using timestamps and trying to get information by date? Easy enough.

Make sure you model has the date as one of the fields so you can limit by date, and sort by time.

Generate a view that fetches x amount of requests, sorted by time, limit by date as a results array. You want to fetch x + 1and store as a field next (if it exists). Output the data in a format that can be parsed, say XML or JSON.

On your server side, create a script that fetches results. If the next variable is passed then fetch again, except you want >= or <= (depending on your sort) for the timestamp in the “next” entry.

Your server side script can then keep fetching until next is not passed, and you have all your results.

I’ve also used this method for deleting large amounts of data using javascript, where I set up a delete button that kicked off a javascript function that would keep requesting to delete until appengine told my script (based on logic I wrote) there were no more to delete.

You comment about having an image in the cloud made me think that my attempt to run Scala compiler inside the GAE Servlet is not just for building an educational site – http://github.com/vydra/gae-scala/tree/master

For doing adhoc quries you could use the remote api. The time limit does not apply here. (But the limits on queru length and offsets unfortunately do – unless you want to resort to custom orderable keys – another huge pain). However, the biggest annoyances I am facing is clearing of the production server. Takes a good couple of days to erase the entire DB. No body has been able to tell me why this has to be the case. Why cant there be a simple clear switch?

cheers
Sro