[#CMP-286] transaction integration prevents the index operation from operating in BATCH_INSERT mode

Compass

transaction integration prevents the index operation from operating in BATCH_INSERT mode

Created: 13/Oct/06 06:05 PM Updated: 21/Oct/06 11:41 AM

Component/s:

Compass::Core

Affects Version/s:

1.1 M2

Fix Version/s:

1.1 M3

Description

« Hide

The index process effectively joins the existing transaction (rather than being able to create a new transaction, as required, w/ isolation level BATCH_INSERT):

See:
1) SingleCompassTemplate.executeForIndex
2) which calls CompassTemplate.execute
3) which calls DefaultCompass.openSession
4) which calls new DefaultCompassSession
5) which calls: transactionFactory.tryJoinExistingTransaction(this);

which effectively creates a new transation with standard semantics.

Afterwards, in step 2(CompassTemplate.execute), it calls session.beginTransaction(transactionIsolation) with an isolation level of BATCH_INSERT; however, that call eventually results in joining the existing thread bound transaction (the one created in step 5 w/ non BATCH_INSERT semantics).

I hope this helps. The probably basically means index operations on large amounts of data basically blow up memory usage.

All

Comments

Change History

Sort Order:

[ Permlink | « Hide ]

Shay Banon added a comment - 14/Oct/06 02:24 PM

Yep, you are right. I added the tryJoin... in order to simplify the usage of sessions within managed transactions environments, and did not think that it will cause the BATCH_INSERT transaction isolation to be ignored.

I have added another setting:

/**

When opening a session, Compass tries to automatically start a transaction
and join it. This might mean that transaction settings when running within
a managed environment won't take affect. The settings allows to disable
the auto joining of a session to a transaction.
*/
public static final String DISABLE_AUTO_JOIN_SESSION = "disableAutoJoinSession";

And the SingleCompsasGps sets it to true. It will cause the BATCH_INSERT isolation to be taken into account. I have tested it locally, and it seems to work. Can you please give 1.1 M3 SNAPSHOT a try and verify it as well?

Nice catch mate! Thanks.

[ Permlink | « Hide ]

Damian Murphy added a comment - 17/Oct/06 01:26 AM

I'm experiencing this exact issue with 1.1 M3 SNAPSHOT.

A search of the 1.1 M3 SNAPSHOT source code doesn't include anything like "DISABLE_AUTO_JOIN_SESSION" anywhere.

[ Permlink | « Hide ]

Shay Banon added a comment - 17/Oct/06 03:00 AM

I just downloded 1.1 M3, and if you have a look at CompassEnvironment class (under org/compass/core/config), you can see the setting (DISABLE_AUTO_JOIN_SESSION).

Also, have a look at AbstractTransactionFactory, and see where it is being read.

[ Permlink | « Hide ]

Damian Murphy added a comment - 17/Oct/06 06:59 PM

Okay - I found the code - that'll teach me to rely on Windows file search.

The only way I can get index creation to work without completely consuming all available memory was to replace:

<property name="compassSettings">
<props>
<prop key="compass.engine.connection">jdbc://lucene_index</prop>
<prop key="compass.transaction.factory">org.compass.spring.transaction.SpringSyncTransactionFactory</prop>
<prop key="compass.engine.maxBufferedDocs">100</prop>
<prop key="compass.engine.mergeFactor">200</prop>
<prop key="compass.transaction.disableAutoJoinSession">true</prop>
</props>
</property>

with:

<property name="compassSettings">
<props>
<prop key="compass.engine.connection">jdbc://lucene_index</prop>
<prop key="compass.transaction.factory">org.compass.core.transaction.LocalTransactionFactory</prop>
<prop key="compass.engine.maxBufferedDocs">100</prop>
<prop key="compass.engine.mergeFactor">200</prop>
<prop key="compass.transaction.disableAutoJoinSession">true</prop>
</props>
</property>

[ Permlink | « Hide ]

Shay Banon added a comment - 18/Oct/06 11:34 AM

You don't have to set the disableAutoJoinSession in the settings, it will be automatically set by the CompassGps in the index method. I have double check the fix, and it looks like it does use BATCH_INSERT transaction isolation.

Damian, what happens when you work with local file system connection and not Jdbc (with SpringSync), does it work? You might have a different problem. I have added better debugging which prints out the transaction isolation when starting a transaction, maybe you could check that it starts a BATCH_INSERT transaction isolation for the Gps indexing.

[ Permlink | « Hide ]

Shay Banon added a comment - 18/Oct/06 11:34 AM

By the way, Randy, have you tried the latest 1.1 M3 SNAPSHOT? Does it fix your problem?

[ Permlink | « Hide ]

Randy Puro added a comment - 18/Oct/06 01:56 PM

For our project, we've actually rolled back to v1.0; however, I just ran the index operation using the latest 1.1M3 SNAPSHOT and it definitely has fixed the problem. Thanks a lot for your help! I'm excited about the new features in 1.1 (particularly the ability to simply use a CompassSession as an injected bean within a transactional scope) and will look forward to getting back to the 1.1 release. Thanks for the good communcation and help.

[ Permlink | « Hide ]

Damian Murphy added a comment - 19/Oct/06 01:55 AM

With the above configuration, including the SpringSyncTransactionFactory and a local filesystem connection (not JDBC) I still get eternal memory consumption leading to out of Java Heap Space.

[ Permlink | « Hide ]

Shay Banon added a comment - 21/Oct/06 11:40 AM

Damina, maybe you could lower the maxBufferedDocs and mergeFactor, they cause the indexing process to consume more memory.

Anyhow, I am going to close this issue, since it seems like the bug it was opened for is fixed. If you still have problems, the forum would be the best place (and of course, if there is a bug, we will fix it under a different issue).

Thanks Randy for this one, I could have missed it and it is a major bug.