You may be interested in looking at GAEBAR (Google App Engine Backup and Restore). I haven’t tried it, but it claims to backup your entire dataset. EVen if it’s not usable directly, it may well contain some hints on how to do so for you.

Thanks for the pointer. That solves the ad hoc analysis problem, but if I want to trial features that happen to need to traverse a lot of data I still can’t do it in AppEngine.

Hi Kent, I was asking you about this on twitter yesterday (I’m @mcobrien). I hadn’t realised you were using Java on appengine, but remote download (and a slow but useful remote REPL) are both possible with the python runtime.

You can actually upload a python version of your app to get access to these features — see this post on the ae group: http://groups.google.com/group/google-appengine/browse_thread/thread/2f31defb815308ee/af9650efeac0666b

Once you have a python version installed, take a look at setting up the remote API:

http://code.google.com/appengine/articles/remote_api.html

Hope this helps, let me know if you have any problems as I’ve done this (via python) a couple of times!

cheers
Michael

Thanks. That will solve my immediate need to mine the data I have so far.

Hey Kent,

Sorry to hear you’ve encountered some bumps along the way. Things can work a little differently in App Engine, but I believe it’s great for small and large applications alike. Comments here have already pointed out some good solutions to your problems; I’d like to point out a couple others:

First, the App Engine request time limit is now *30 seconds*. It was 10 seconds at launch in April 2008. You can do a full 30 second burn of CPU usage in a single request to accommodate the type of work you’ve described.

You can definitely do *online* ad hoc queries on the production data (without downloading it), as long as your query does not require a new index to be built. You can use the Datastore Viewer in the admin console to do this or add a shell/repl app to run arbitrary code that you type in. See an example shell here:
http://lotrepls.appspot.com/

Ad hoc queries can be satisfied primarily with the “merge-join” type of query, which lets you do multiple equality filters in a single query. This can satisfy set membership tests (like “How many people are setting their user ids?”) very easily. This concept is covered in detail in this Google I/O talk (the video should be online today or very soon):
http://code.google.com/events/io/sessions/BuildingScalableComplexApps.html

Otherwise, for background work like schema migration and modifying a whole sequence of data, we are soon launching our Task Queue API. It’s described in detail in this Google I/O talk (which should also have its video online sometime today or very soon):
http://code.google.com/events/io/sessions/OfflineProcessingAppEngine.html

Hope that helps,

-Brett

Also, the time limits have been significantly raised.

http://code.google.com/appengine/docs/python/runtime.html

(They are at the bottom of the page)

Brett,

It’s good to know someone there is listening. The changes you mention will help with some of my concerns, especially mining data for patterns that can be used as the seeds of features.

The big remaining problem for me is performance. If I have < 100MB of data that could easily be represented in memory as POJOs, it seems like there is at least a two order of magnitude difference in performance between working through JDO and working in memory (I should measure this). If I am experimenting quickly, a feature could take 5 minutes to code against in-memory data and a day to code going through the database. As a bootstrap startup searching for the thread that will begin unraveling the revenue ball, this is an enormous difference. A higher timeout limit won’t fix this–I get better market feedback from a snappy service than a sluggish one.

Kent,

I think its a reasonable strategy to use GAE to collect data since scalability needs arise at the most inconvenient time . Once collected, transfer the data to Amazon EC2 and only pay for storage and CPU time of the actual analysis. They even have http://aws.amazon.com/elasticmapreduce now. I don’t know if there are already libraries to help with this, but even a hand-rolled solution should be very small.

I was coming to something like that as well. It is still more engineering investment than I would like to be making given the infant mortality rate of my features. What’s frustrating is that there isn’t a good dynamic, in-memory solution. I keep going back to Smalltalk–if I could just have a Smalltalk image live on the web I could do all the experimenting I want five minutes of coding at a time. When the data started pushing 300MB I could either go to Gemstone or do pre-processing as necessary for performance in a more conventional database technology.

Hey Kent,

There is memcache support right now available through JCache:
http://code.google.com/appengine/docs/java/memcache/usingjcache.html

This gives you a high-speed, in-memory cache for accessing frequently used objects. You can insert POJOs in here as long as they can serialize. I’m not sure which frameworks support transparent caching of JDO queries, but this seems straight forward to add for your needs.

Otherwise, you should take another read through the Google Bigtable research paper. It describes how data is cached in memory when it’s frequently accessed. This is what gives the App Engine Datastore such good read performance. That doc is here:
http://labs.google.com/papers/bigtable.html

Combining these two together (memcache and Bigtable caching through our Datastore) you can get some serious read performance from App Engine.

-Brett

ps: Looks like you didn’t moderate my first comment to be visible to everyone. =)

Brett,

Not approving your previous comment was a complete oversight on my part. No intention intended.

Thanks for the additional information about performance tuning. I have read the BigTable paper a couple of times. However, I am not seeing delivered performance by the time I get through all the layers. I have what look to me like modest queries that fail intermittently.

Again, I’d rather not spend time engineering this. I’d rather have a bunch of objects in memory. I know how to engineer that for performance and I’m willing to take the hit scaling later. However, I can imagine I am not your target audience. It could be that I just need to bring my performance tuning skills into the 21st century.

Regards (and I mean that ,

Kent

There’s also Java REPL options – http://googleappengine.blogspot.com/2009/04/many-languages-and-in-runtime-bind-them.html

Doh – didn’t refresh before I posted the previous comment!

Hi,

For doing adhoc quries you could use the remote api. The time limit does not apply here. (But the limits on queru length and offsets unfortunately do – unless you want to resort to custom orderable keys – another huge pain). However, the biggest annoyances I am facing is clearing of the production server. Takes a good couple of days to erase the entire DB. No body has been able to tell me why this has to be the case. Why cant there be a simple clear switch?

cheers
Sro

Kent,

You comment about having an image in the cloud made me think that my attempt to run Scala compiler inside the GAE Servlet is not just for building an educational site – http://github.com/vydra/gae-scala/tree/master

I too was going to ask whether in-memory caching (namely memcache) would help with the data traversal and time limit problems. Memcache isn’t as sophisticated as, say, Oracle Coherence but it’s better than the quite limited persistence layer that GAE has with DataNucleus.

I think you need to think outside of the box a little on this. You need to understand appengine is a bunch of servers that you can create java/python applications in, and it has a datastore backend. Most requests are done via HTTP. It also has time constraints for how long a request can run.

With these limitations in mind, you need to do some development in order to get at your data reliably.

First, since you’re pulling logs, I assume you’re using timestamps and trying to get information by date? Easy enough.

Make sure you model has the date as one of the fields so you can limit by date, and sort by time.

Generate a view that fetches x amount of requests, sorted by time, limit by date as a results array. You want to fetch x + 1and store as a field next (if it exists). Output the data in a format that can be parsed, say XML or JSON.

On your server side, create a script that fetches results. If the next variable is passed then fetch again, except you want >= or <= (depending on your sort) for the timestamp in the “next” entry.

Your server side script can then keep fetching until next is not passed, and you have all your results.

I’ve also used this method for deleting large amounts of data using javascript, where I set up a delete button that kicked off a javascript function that would keep requesting to delete until appengine told my script (based on logic I wrote) there were no more to delete.

Kent, I dropped you a tweet but wanted to explain in more detail…
Amazon EC2 is a very good option for building cloud-based scalable Java apps. Raw EC2 is more complicated and more pricey than GAE, but more flexible and supports standard Java+DB apps. If you’re looking for your “lightning speed” dev workflow, Stax.net is a Java Platform-as-a-Service for building and deploying apps on EC2 that provides Java developers with a similar create/dev/deploy workflow, but without many of the GAE restrictions. Like GAE, Stax.net is still in beta, but you get a hefty load of EC2 goodness for free while you are building your apps.

I’m aware of EC2. It isn’t the technology platform in AppEngine that forms the bottleneck, it’s the difficulty in experimenting.

Even Java isn’t what I’m looking for. What I’d like is a maximum of 100MB of easily-persistable objects that can be programmed very flexibly and (because they are all in memory) very efficiently. Maybe JavaScript/V8 on the server and/or an in-memory version of Map/Reduce. I’m afraid I’m not terribly articulate about this, because it’s just a feeling at this point. I’ve been through this enough times to know that I may be just beginning to uncover something very interesting or not.

AppEngine does a brilliant job of its location on the scaling/flexibility tradeoff, I’m just looking for different spot.

Hi Kent;

You discuss an area in your post that I am stuck on. I am fluent in web app development in Java. And I am evaluating GAE for my strt-up too. My choice for sticking with GAE is only its giving me managed hosting for some popular Java web frameworks (I will use Struts). I dont want to mess with a naked server and putty it all the day in solving configuration issues in my web app. So this my motivation behind thinking about GAE.

So far I have code some servlets in it to understand its environment. But most concerned are for me is DB. I am unable to find any Java code snippet about interacting with Database. They are not supporting relational database – that increases my concern about data. How I will see my data or as you said how I will run queries on it.

I will love to hear if you point me towards some better understanding of DB stuff. Is there any tutorial or some code snippet?

Comments are welcomed.

To learn AppEngine’s database I worked through http://code.google.com/appengine/docs/java/datastore/ several times. I’d read it through, add a little functionality, read it again, add a little more. Also, I found the local testing mode helpful for speeding up the learning cycle, especially when used with TDD. Copying from code that was out there in bits and pieces, I use this fixture to write tests that can read and write data:

@Before
public void setUp() throws Exception {
ApiProxy.setEnvironmentForCurrentThread(new TestEnvironment());
ApiProxy.setDelegate(new ApiProxyLocalImpl(new File(".")){});
ApiProxyLocalImpl proxy = (ApiProxyLocalImpl) ApiProxy.getDelegate();
proxy.setProperty(LocalDatastoreService.NO_STORAGE_PROPERTY, Boolean.TRUE.toString());
}

@After
public void tearDown() throws Exception {
ApiProxyLocalImpl proxy = (ApiProxyLocalImpl) ApiProxy.getDelegate();
LocalDatastoreService datastoreService = (LocalDatastoreService) proxy.getService("datastore_v3");
datastoreService.clearProfiles();
ApiProxy.setDelegate(null);
ApiProxy.setEnvironmentForCurrentThread(null);
}

[...] profile to Facebook without manually retyping everything.” Another report entitled “AppEngine doesn’t fit the needs of startups on the runway” described how limited data access made impossible to run a statistical analysis of the user [...]