|
[
Permlink
| « Hide
]
Patrick Twohig added a comment - 01/Jan/10 02:16 AM
Maybe I'm Doing It Wrong, but here's a bit of code that reproduces the bug. From what I can read in the GoogleAppEngineDirectory class, it starts manipulating the data store within the context of a transaction, however the index isn't stored in the same entity group as the object being stored to the data store. In this case it's the instance of TestDataObject.
Here's the patch! I ended up using the latest Google App Engine API. A few things have changed since the original directory store has been written, which appears to be the ability to run queries from within a transaction. I also had to add a few lines to some of the unit test code, but I ran through the tests a few times and they all appear to have passed.
Here's what this patch does and the rationale behind it all. 1) It puts all locks in their own entity group, separate from the entities which house the index. This way a separate transaction can be used to lock/unlock the search index while writing. According to the guys in the #appengine IRC channel and from what I could gather form the GAE documentation putting two entities in the same entity group isn't necessary unless you want to use them within the same transaction. In short it provides not significant performance improvements. In our case, with locks, we're doing exactly the opposite and never manipulate a lock outside of a transaction so there's no good reason to put it in the same entity group as the directory. Happy New Year, Maybe I applied the patch wrong and/or built the jar wrong, or maybe this is a separate issue totally...
I get the following error when persisting objects to the datastore: .... com.bitdual.client.rpc.RedirectException' threw an unexpected exception: org.compass.gps.device.jdo.JdoGpsDeviceException: {appengine}: Failed while updating [1:1]; nested exception is org.compass.core.engine.SearchEngineException: Transaction is set as read only at com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:378) at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:581) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:188) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:224) at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1093) at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:51) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:121) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) at com.google.apphosting.utils.jetty.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:70) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:352) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:313) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:506) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:844) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:644) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:396) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442) Caused by: org.compass.gps.device.jdo.JdoGpsDeviceException: {appengine}: Failed while updating [1:1]; nested exception is org.compass.core.engine.SearchEngineException: Transaction is set as read only at org.compass.gps.device.jdo.Jdo2GpsDevice$JdoGpsInstanceLifecycleListener.postStore(Jdo2GpsDevice.java:158) at org.datanucleus.jdo.JDOCallbackHandler.postStore(JDOCallbackHandler.java:140) at org.datanucleus.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3189) at org.datanucleus.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3161) at org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1298) at org.datanucleus.ObjectManagerImpl.persistObject(ObjectManagerImpl.java:1175) at org.datanucleus.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:669) at org.datanucleus.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:694) at com.bitdual.server.dao.QuestionDao.save(QuestionDao.java:271) at com.bitdual.server.QuestionServiceImpl.save(QuestionServiceImpl.java:102) at com.bitdual.server.QuestionServiceImpl.save(QuestionServiceImpl.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.google.appengine.tools.development.agent.runtime.Runtime.invoke(Runtime.java:100) at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:562) ... 30 more Caused by: org.compass.core.engine.SearchEngineException: Transaction is set as read only at org.compass.core.lucene.engine.LuceneSearchEngine.verifyNotReadOnly(LuceneSearchEngine.java:130) at org.compass.core.lucene.engine.LuceneSearchEngine.createOrUpdate(LuceneSearchEngine.java:273) at org.compass.core.lucene.engine.LuceneSearchEngine.save(LuceneSearchEngine.java:263) at org.compass.core.impl.DefaultCompassSession.save(DefaultCompassSession.java:489) at org.compass.core.impl.DefaultCompassSession.save(DefaultCompassSession.java:473) at org.compass.core.impl.ExistingCompassSession.save(ExistingCompassSession.java:313) at org.compass.core.impl.ExistingCompassSession.save(ExistingCompassSession.java:313) at org.compass.gps.device.jdo.Jdo2GpsDevice$JdoGpsInstanceLifecycleListener$2.doInCompassWithoutResult(Jdo2GpsDevice.java:151) at org.compass.core.CompassCallbackWithoutResult.doInCompass(CompassCallbackWithoutResult.java:29) at org.compass.core.CompassTemplate.execute(CompassTemplate.java:133) at org.compass.gps.impl.SingleCompassGps.executeForMirror(SingleCompassGps.java:151) at org.compass.gps.device.jdo.Jdo2GpsDevice$JdoGpsInstanceLifecycleListener.postStore(Jdo2GpsDevice.java:149) ... 46 more Dumb question, but did this happen only after you applied the patch? The stack trace doesn't indicate that the org.compass.needle.gae.* classes directly caused the exception. If the patch is the cause of the problem, my best guess is that after GoogleAppEngineDirectory.doInTransaction() is called, it leaves the DatastoreService object in a screwed up state.
What I do know is that the call to PersistenceManager.makePersistent(Object) calls some Compass code which doesn't necessarily play well with how GAE deals with transactions. What happens in LuceneSearchEngine.createOrUpdate()? It happened both before and after I applied the patch.
I'm working on a test app in hopes of isolating the issue, whether it be in your patch, compass, and/or my code(the latter two being more likely, and that the last most likely). Some combination of your patch and the following change in my code resolved my issue.
CompassSearchSession search = PMF.getCompass().openSearchSession(); Ah, well, that's good. It's always best practice to close what you leave open...now that I think about it there may be places where I forget to do that in my code....anyhow. My best guess is that without calling close() you leave some write-lock in place which screws up the index the next time you try to access it.
Hi Patrick and Shay,
First of all, nice work! I wonder if the following approach works, which looks a little bit simpler to me: 1) Move the mirroring operations of a GpsDevice to the GAE background tasks and push these tasks to a task queue. So if the current write operation to datastore is in a transaction conext, this transaction will not be interfered since no index, which resides in a different entity group, is written yet. 2) When GAE runs the tasks (using a new thread), wrap the mirroring operations by a transaction to ensure isolation between index writes. This transaction includes only one sigle entity group that the index belongs to. The drawback of this approach is that the order by which the index is written may differ from that by which the domain objects are saved, since the GAE queue may not execute tasks in FIFO order all the time (for example, when GAE system crashed and is recovering). But this may be fine to some applications (at least mine Any comment? Changing the title to reflect the changes required.
Committed the changes, thanks, they look good. Hopefully this will solve the transaction problems people were having.
Let's hope so. I think pinecone is on to something. It's probably a good idea to offload the indexing into a separate task. That being said, it's probably a good idea to file that under another issue if anybody makes any headway on it.
Patrick,
Yes, I spent some time on my proposal, and the out-of-order problem can be solved by using versioning (i.e., @version in JPA/JDO). However, later I realized that there is a limitation on GAE: all tasks in different queues together can have execution rate at most 20/sec. I think currently, your patch is good enough I think task queueing is the best solution because we may be over burdening the transaction every time. But I'd say wait until task queues for Java are no longer in the "experimental" stage.
Hello,
I still seem to be experiencing index corruption. From my understanding, what these improvements have done is to isolate updates to index entities made from any pre-existing transactions and hence avoid the problem of updating entities in multiple entity groups in a transaction which is not allowed in App Engine. However, correct me if I'm wrong but changes made to index entities within an index session even for a single save operation may be done by multiple write operations which would each run in their own transactions. So does this mean that if I'm doing a save operation and there are some writes committed but eventually there is a write error, for example due to App Engines annoying 30 second limit being reached, then the index would be corrupted? Len Hello,
I'm encountering similar problem with transactions. Please note that I'm not using automatic indexes and I'm doing indexing manually, i.e indexing is executed by task queue call page by page until all entities where indexed. When any of my entities is updated, new task is added in the task queue for that entity. Here is the error which I'm encountering when multiple users are editing multiple documents: org.compass.core.engine.SearchEngineException: Failed to prepare transaction for sub index [contractentity]; nested exception is java.lang.NullPointerException: null I just started having a similar issue last night. None of the classes in your stack trace pertain to what's in the patch, maybe file under a different issue?
|
||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Resolved
Major
Google App Engine: Improved transaction management
)