[openjms-developer] Fw: MOM replay management : OpenJMS MOM 0.7.5.

Régis LEBRUN

2003-10-09 16:06:56 UTC

I received a mail error sending my email to my previews contacts on OpenJMS. I don't know why.

please find here after a brief description of our problem.
I hope you will be able to help us

Kind Regards
Régis Lebrun

----- Original Message -----
From: Régis LEBRUN
To: ***@intalio.com ; ***@intalio.com
Cc: Julien Schreiber
Sent: Thursday, October 09, 2003 5:23 PM
Subject: MOM replay management : OpenJMS MOM 0.7.5.

Hi,

Julien, a guy working in my team sent you an email last week. As we didn't receive any news from you and as the problem is quite important (lost of messages), I wanted to bring you some precisions. Please, we need a quick answer.

The problem is systematically reproduced when we use a listener on another server (physical hardware implementation) than the MOM
It is not due to a huge traffic.

We use server A and server B.
If we install the MOM on server A and the listener in server B, we have the problem (100 messages lost for 54 000 messages sent at a frequency of 5 messages per second)
if we install the MOM on server A and the listener also, we never have the problem in hte same load configuration.

What we think l is that it can be a socket lost that is badly managed by the MOM.

Could you please have a look on that and give me a status.

Thank's a lot

Regis Lebrun

----- Original Message -----
From: Julien Schreiber
To: ***@intalio.com ; ***@intalio.com
Cc: Régis Lebrun
Sent: Monday, September 29, 2003 1:52 PM
Subject: MOM replay management

Hi,

I am currently using OpenJMS MOM 0.7.5.

I have already asked some questions a few month ago that appear below.

Here is my new question : I would like to know if it is possible to tune the MOM in order to try to send again messages that have not been transmitted correctly through the MOM. It seems that with the MOM I use, if too many messages are passed in the MOM at the same time, 2 kinds of problems can occur :
- Messages are lost. No error messages or exception are caught that indicate that a problem occured. So everything seems to be done correctly from client side point of view.
- Messages take 4 seconds to be transmitted through the MOM (it normally takes a few milliseconds when everything goes well). Once again, everything seems to be done correctly from client side point of view: Messages are "sent", and there is no information that prove that queues are saturated.

I have run load tests at 15 requests per second during a few minutes. The message passed is a String. A dozen requests are lost or take a few seconds to be transmitted through the MOM (4seconds in the MOM)

Here above is the current configuration of the MOM :

************************************
<?xml version="1.0"?>



<Configuration>


<ServerConfiguration host="172.28.20.131" embeddedJNDI="true" />


<Connectors>
<Connector scheme="rmi">
<ConnectionFactories>
<QueueConnectionFactory name="JmsQueueConnectionFactory" />
</ConnectionFactories>
</Connector>
</Connectors>















<DatabaseConfiguration>
<JdbmDatabaseConfiguration name="openjms.db" />
</DatabaseConfiguration>


<AdminConfiguration
script="${openjms.home}/../../scripts/mom.sh -start"
config="${openjms.home}/../MomServer.xml" />


<AdministeredDestinations>
<AdministeredQueue name="requestLocQueue" />
<AdministeredQueue name="locResultQueueA" />
<AdministeredQueue name="locResultQueueB" />
<AdministeredQueue name="isAliveQueue" />
</AdministeredDestinations>

</Configuration>

************************************

Thank in advance for your answer and tips to tune correctly the MOM

Regards,

Julien Schreiber
Cap Gemini Telecom France

----- Original Message -----
From: Jim Alateras
To: 'Julien Schreiber'
Cc: 'alexis marchand' ; 'Régis Lebrun'
Sent: Tuesday, March 25, 2003 12:12 AM
Subject: RE: Problems with version 0.7.3.1 and 0.7.4

JDBM database should not be used for load testing and is only distributed for quick out-of-the-box setup and testing. You should really choose one of RDBMS databases (i.e. mysql, hsql, oracle etc) for testing. With RDBMS we have been able to get 100's of messages per second (size of message 1KB).

You should also be using 0.7.4 or wait for the latest version 0.7.5 to be released.

If your persistent message requirements exceed 200x1KB messages/second then you may need to source a new JMS provider. However anything within the 200x1KB messages/second should be achievable with OpenJMS.

cheers

</jima>

-----Original Message-----
From: Julien Schreiber [mailto:***@cgey.com]
Sent: Monday, March 24, 2003 11:24 PM
To: ***@intalio.com
Cc: alexis marchand; Régis Lebrun
Subject: Problems with version 0.7.3.1 and 0.7.4

Hi,

We have tried to use the new version of the OpenJMS MOM : 0.7.4 on Solaris.

Performances are not as good as it was with version 0.7.3.1 : Processing Time is longer than before.

We have run a test with a frequency of 3 messages sent in the MOM per second.

Here is the configuration we used :

***********************************************************************************************************************************

<?xml version="1.0"?>



<Configuration>


<ServerConfiguration host="10.67.178.28" embeddedJNDI="true" />


<Connectors>
<Connector scheme="rmi">
<ConnectionFactories>
<QueueConnectionFactory name="JmsQueueConnectionFactory" />
</ConnectionFactories>
</Connector>
</Connectors>


<LeaseManagerConfiguration sleepTime="300" />


<MessageManagerConfiguration destinationCacheSize="10" />


<SchedulerConfiguration maxThreads="10" />


<GarbageCollectionConfiguration memoryCheckInterval="60"
lowWaterThreshold="20"
garbageCollectionInterval="120"
garbageCollectionThreadPriority="5">
</GarbageCollectionConfiguration>


<DatabaseConfiguration>
<JdbmDatabaseConfiguration name="openjms.db" />
</DatabaseConfiguration>


<AdminConfiguration
script="${openjms.home}/../../scripts/mom.sh -start"
config="${openjms.home}/../MomServer.xml" />


<AdministeredDestinations>
<AdministeredQueue name="requestLocQueue" />
<AdministeredQueue name="locResultQueueA" />
<AdministeredQueue name="locResultQueueB" />
<AdministeredQueue name="isAliveQueue" />
</AdministeredDestinations>

</Configuration>

***********************************************************************************************************************************

We had also endurance problems with version 0.7.3.1

Here is the error messages we had during a Load Test with a frequency of 5 messages sent in the MOM per second. Error occured 1 hour after the begining of the Test :

***********************************************************************************************************************************

13:49:51.636 ERROR [Scheduler-Worker-3] - Dropping handle Handle : 0:1048427372219:1494979:ID:15591976861184663 since we cannot resolve it.
13:49:51.652 ERROR [Scheduler-Worker-3] - javax.jms.JMSException: Could not get message for handle Handle : 0:1048427372219:1494979:ID:15591976861184663
javax.jms.JMSException: Could not get message for handle Handle : 0:1048427372219:1494979:ID:15591976861184663
at org.exolab.jms.server.JmsServerSession.onMessage(Unknown Source)
at org.exolab.jms.messagemgr.QueueConsumerEndpoint.run(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runIt(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runWork(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker.access$000(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:479)
15:23:51.466 ERROR [Scheduler-Worker-4] - Dropping handle Handle : 0:1048433014999:1548927:ID:7739832439311948131 since we cannot resolve it.
15:23:51.467 ERROR [Scheduler-Worker-4] - javax.jms.JMSException: Could not get message for handle Handle : 0:1048433014999:1548927:ID:7739832439311948131
javax.jms.JMSException: Could not get message for handle Handle : 0:1048433014999:1548927:ID:7739832439311948131
at org.exolab.jms.server.JmsServerSession.onMessage(Unknown Source)
at org.exolab.jms.messagemgr.QueueConsumerEndpoint.run(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runIt(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runWork(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker.access$000(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:479)
02:09:59.757 ERROR [Scheduler-Worker-7] - Dropping handle Handle : 0:1048471796879:1919720:ID:6589455389744779712 since we cannot resolve it.
02:09:59.758 ERROR [Scheduler-Worker-7] - javax.jms.JMSException: Could not get message for handle Handle : 0:1048471796879:1919720:ID:6589455389744779712
javax.jms.JMSException: Could not get message for handle Handle : 0:1048471796879:1919720:ID:6589455389744779712
at org.exolab.jms.server.JmsServerSession.onMessage(Unknown Source)
at org.exolab.jms.messagemgr.QueueConsumerEndpoint.run(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runIt(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runWork(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker.access$000(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:479)

***********************************************************************************************************************************

Here is the configuration we used :

***********************************************************************************************************************************

<?xml version="1.0"?>



<Configuration>


<ServerConfiguration host="10.67.178.28" embeddedJNDI="true" />


<Connectors>
<Connector scheme="rmi">
<ConnectionFactories>
<QueueConnectionFactory name="JmsQueueConnectionFactory" />
</ConnectionFactories>
</Connector>
</Connectors>


<LeaseManagerConfiguration sleepTime="300" />


<MessageManagerConfiguration destinationCacheSize="10" />


<SchedulerConfiguration maxThreads="10" />


<GarbageCollectionConfiguration memoryCheckInterval="60"
lowWaterThreshold="20"
garbageCollectionInterval="120"
garbageCollectionThreadPriority="5">
</GarbageCollectionConfiguration>


<DatabaseConfiguration>
<JdbmDatabaseConfiguration name="openjms.db" />
</DatabaseConfiguration>


<AdminConfiguration
script="${openjms.home}/../../scripts/mom.sh -start"
config="${openjms.home}/../MomServer.xml" />


<AdministeredDestinations>
<AdministeredQueue name="requestLocQueue" />
<AdministeredQueue name="locResultQueueA" />
<AdministeredQueue name="locResultQueueB" />
<AdministeredQueue name="isAliveQueue" />
</AdministeredDestinations>

</Configuration>

***********************************************************************************************************************************

Could you tell us if the configuration we use could be appplied to the 2 versions ? Could you give us some tips to customize the MOM to increase performances and endurance ? Could you give us some information on changes between the 2 versions explaining the bad performances we saw while running the Test ?

We have 3 days to find a solution regarding both the load performances and the endurance of the MOM.

Then we will have to change the OpenJMS MOM to another one.

Regards,

___________________________
Julien Schreiber
Cap Gemini Telecom Media & Networks
+33 (0)1 49 00 47 60

Tim Anderson

2003-10-09 22:46:40 UTC

Permalink

Your configuration indicates you are using the JDBM database.
While I'm not aware of it dropping messages, this database should
not be used for anything other than basic evaluation of OpenJMS.
It will be a major limiting factor for message thoughput.

As Jim mentioned in one of his replies, you should be using an RBDMS
such as MySQL, Sybase, or Oracle.
-----Original Message-----
From: openjms-developer-***@lists.sourceforge.net
[mailto:openjms-developer-***@lists.sourceforge.net]On Behalf Of Régis
LEBRUN
Sent: Friday, 10 October 2003 2:07 AM
To: openjms-***@lists.sourceforge.net
Cc: Julien Schreiber
Subject: [openjms-developer] Fw: MOM replay management : OpenJMS MOM
0.7.5.

I received a mail error sending my email to my previews contacts on
OpenJMS. I don't know why.

please find here after a brief description of our problem.
I hope you will be able to help us

Kind Regards
Régis Lebrun

----- Original Message -----
From: Régis LEBRUN
To: ***@intalio.com ; ***@intalio.com
Cc: Julien Schreiber
Sent: Thursday, October 09, 2003 5:23 PM
Subject: MOM replay management : OpenJMS MOM 0.7.5.

Hi,

Julien, a guy working in my team sent you an email last week. As we didn't
receive any news from you and as the problem is quite important (lost of
messages), I wanted to bring you some precisions. Please, we need a quick
answer.

The problem is systematically reproduced when we use a listener on another
server (physical hardware implementation) than the MOM
It is not due to a huge traffic.

We use server A and server B.
If we install the MOM on server A and the listener in server B, we have
the problem (100 messages lost for 54 000 messages sent at a frequency of 5
messages per second)
if we install the MOM on server A and the listener also, we never have the
problem in hte same load configuration.

What we think l is that it can be a socket lost that is badly managed by
the MOM.

Could you please have a look on that and give me a status.

Thank's a lot

Regis Lebrun

----- Original Message -----
From: Julien Schreiber
To: ***@intalio.com ; ***@intalio.com
Cc: Régis Lebrun
Sent: Monday, September 29, 2003 1:52 PM
Subject: MOM replay management

Hi,

I am currently using OpenJMS MOM 0.7.5.

I have already asked some questions a few month ago that appear below.

Here is my new question : I would like to know if it is possible to tune
the MOM in order to try to send again messages that have not been
transmitted correctly through the MOM. It seems that with the MOM I use, if
too many messages are passed in the MOM at the same time, 2 kinds of
problems can occur :
- Messages are lost. No error messages or exception are caught
that indicate that a problem occured. So everything seems to be done
correctly from client side point of view.
- Messages take 4 seconds to be transmitted through the MOM (it
normally takes a few milliseconds when everything goes well). Once again,
everything seems to be done correctly from client side point of view:
Messages are "sent", and there is no information that prove that queues are
saturated.

I have run load tests at 15 requests per second during a few minutes.
The message passed is a String. A dozen requests are lost or take a few
seconds to be transmitted through the MOM (4seconds in the MOM)

Here above is the current configuration of the MOM :

************************************
<?xml version="1.0"?>



<Configuration>


<ServerConfiguration host="172.28.20.131" embeddedJNDI="true" />


<Connectors>
<Connector scheme="rmi">
<ConnectionFactories>
<QueueConnectionFactory name="JmsQueueConnectionFactory" />
</ConnectionFactories>
</Connector>
</Connectors>















<DatabaseConfiguration>
<JdbmDatabaseConfiguration name="openjms.db" />
</DatabaseConfiguration>


<AdminConfiguration
script="${openjms.home}/../../scripts/mom.sh -start"
config="${openjms.home}/../MomServer.xml" />


<AdministeredDestinations>
<AdministeredQueue name="requestLocQueue" />
<AdministeredQueue name="locResultQueueA" />
<AdministeredQueue name="locResultQueueB" />
<AdministeredQueue name="isAliveQueue" />
</AdministeredDestinations>

</Configuration>

************************************

Thank in advance for your answer and tips to tune correctly the MOM

Regards,

Julien Schreiber
Cap Gemini Telecom France

----- Original Message -----
From: Jim Alateras
To: 'Julien Schreiber'
Cc: 'alexis marchand' ; 'Régis Lebrun'
Sent: Tuesday, March 25, 2003 12:12 AM
Subject: RE: Problems with version 0.7.3.1 and 0.7.4

JDBM database should not be used for load testing and is only
distributed for quick out-of-the-box setup and testing. You should really
choose one of RDBMS databases (i.e. mysql, hsql, oracle etc) for testing.
With RDBMS we have been able to get 100s of messages per second (size of
message 1KB).

You should also be using 0.7.4 or wait for the latest version 0.7.5 to
be released.

If your persistent message requirements exceed 200x1KB messages/second
then you may need to source a new JMS provider. However anything within the
200x1KB messages/second should be achievable with OpenJMS.

cheers

</jima>

-----Original Message-----
From: Julien Schreiber [mailto:***@cgey.com]
Sent: Monday, March 24, 2003 11:24 PM
To: ***@intalio.com
Cc: alexis marchand; Régis Lebrun
Subject: Problems with version 0.7.3.1 and 0.7.4

Hi,

We have tried to use the new version of the OpenJMS MOM : 0.7.4 on
Solaris.

Performances are not as good as it was with version 0.7.3.1 :
Processing Time is longer than before.

We have run a test with a frequency of 3 messages sent in the MOM per
second.

Here is the configuration we used :

****************************************************************************
*******************************************************

<?xml version="1.0"?>



<Configuration>


<ServerConfiguration host="10.67.178.28" embeddedJNDI="true" />


<Connectors>
<Connector scheme="rmi">
<ConnectionFactories>
<QueueConnectionFactory name="JmsQueueConnectionFactory" />
</ConnectionFactories>
</Connector>
</Connectors>


<LeaseManagerConfiguration sleepTime="300" />


<MessageManagerConfiguration destinationCacheSize="10" />


<SchedulerConfiguration maxThreads="10" />


<GarbageCollectionConfiguration memoryCheckInterval="60"
lowWaterThreshold="20"
garbageCollectionInterval="120"
garbageCollectionThreadPriority="5">
</GarbageCollectionConfiguration>


<DatabaseConfiguration>
<JdbmDatabaseConfiguration name="openjms.db" />
</DatabaseConfiguration>


<AdminConfiguration
script="${openjms.home}/../../scripts/mom.sh -start"
config="${openjms.home}/../MomServer.xml" />


<AdministeredDestinations>
<AdministeredQueue name="requestLocQueue" />
<AdministeredQueue name="locResultQueueA" />
<AdministeredQueue name="locResultQueueB" />
<AdministeredQueue name="isAliveQueue" />
</AdministeredDestinations>

</Configuration>

****************************************************************************
*******************************************************

We had also endurance problems with version 0.7.3.1

Here is the error messages we had during a Load Test with a frequency
of 5 messages sent in the MOM per second. Error occured 1 hour after the
begining of the Test :

****************************************************************************
*******************************************************

13:49:51.636 ERROR [Scheduler-Worker-3] - Dropping handle Handle :
0:1048427372219:1494979:ID:15591976861184663 since we cannot resolve it.
13:49:51.652 ERROR [Scheduler-Worker-3] - javax.jms.JMSException:
Could not get message for handle Handle :
0:1048427372219:1494979:ID:15591976861184663
javax.jms.JMSException: Could not get message for handle Handle :
0:1048427372219:1494979:ID:15591976861184663
at org.exolab.jms.server.JmsServerSession.onMessage(Unknown
Source)
at org.exolab.jms.messagemgr.QueueConsumerEndpoint.run(Unknown
Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runIt(Unknown
Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runWork(Unknown
Source)
at
org.exolab.core.threadPool.ThreadPoolWorker.access$000(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker$1.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:479)
15:23:51.466 ERROR [Scheduler-Worker-4] - Dropping handle Handle :
0:1048433014999:1548927:ID:7739832439311948131 since we cannot resolve it.
15:23:51.467 ERROR [Scheduler-Worker-4] - javax.jms.JMSException:
Could not get message for handle Handle :
0:1048433014999:1548927:ID:7739832439311948131
javax.jms.JMSException: Could not get message for handle Handle :
0:1048433014999:1548927:ID:7739832439311948131
at org.exolab.jms.server.JmsServerSession.onMessage(Unknown
Source)
at org.exolab.jms.messagemgr.QueueConsumerEndpoint.run(Unknown
Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runIt(Unknown
Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runWork(Unknown
Source)
at
org.exolab.core.threadPool.ThreadPoolWorker.access$000(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker$1.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:479)
02:09:59.757 ERROR [Scheduler-Worker-7] - Dropping handle Handle :
0:1048471796879:1919720:ID:6589455389744779712 since we cannot resolve it.
02:09:59.758 ERROR [Scheduler-Worker-7] - javax.jms.JMSException:
Could not get message for handle Handle :
0:1048471796879:1919720:ID:6589455389744779712
javax.jms.JMSException: Could not get message for handle Handle :
0:1048471796879:1919720:ID:6589455389744779712
at org.exolab.jms.server.JmsServerSession.onMessage(Unknown
Source)
at org.exolab.jms.messagemgr.QueueConsumerEndpoint.run(Unknown
Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runIt(Unknown
Source)
at org.exolab.core.threadPool.ThreadPoolWorker.runWork(Unknown
Source)
at
org.exolab.core.threadPool.ThreadPoolWorker.access$000(Unknown Source)
at org.exolab.core.threadPool.ThreadPoolWorker$1.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:479)

****************************************************************************
*******************************************************

Here is the configuration we used :

****************************************************************************
*******************************************************

<?xml version="1.0"?>



<Configuration>


<ServerConfiguration host="10.67.178.28" embeddedJNDI="true" />


<Connectors>
<Connector scheme="rmi">
<ConnectionFactories>
<QueueConnectionFactory name="JmsQueueConnectionFactory" />
</ConnectionFactories>
</Connector>
</Connectors>


<LeaseManagerConfiguration sleepTime="300" />


<MessageManagerConfiguration destinationCacheSize="10" />


<SchedulerConfiguration maxThreads="10" />


<GarbageCollectionConfiguration memoryCheckInterval="60"
lowWaterThreshold="20"
garbageCollectionInterval="120"
garbageCollectionThreadPriority="5">
</GarbageCollectionConfiguration>


<DatabaseConfiguration>
<JdbmDatabaseConfiguration name="openjms.db" />
</DatabaseConfiguration>


<AdminConfiguration
script="${openjms.home}/../../scripts/mom.sh -start"
config="${openjms.home}/../MomServer.xml" />


<AdministeredDestinations>
<AdministeredQueue name="requestLocQueue" />
<AdministeredQueue name="locResultQueueA" />
<AdministeredQueue name="locResultQueueB" />
<AdministeredQueue name="isAliveQueue" />
</AdministeredDestinations>

</Configuration>

****************************************************************************
*******************************************************

Could you tell us if the configuration we use could be appplied to the
2 versions ? Could you give us some tips to customize the MOM to increase
performances and endurance ? Could you give us some information on changes
between the 2 versions explaining the bad performances we saw while running
the Test ?

We have 3 days to find a solution regarding both the load performances
and the endurance of the MOM.

Then we will have to change the OpenJMS MOM to another one.

Regards,

___________________________
Julien Schreiber
Cap Gemini Telecom Media & Networks
+33 (0)1 49 00 47 60