What is The CMFActivity Tool

The CMFActivity Tool allows to start methods and scripts in background. If you need to run the same method on many objects, this is not good to start a single transaction and modify all of them. If you do so, then a very long transaction may be required and there is a very high chance that you will get locks if the zope server is under use.

With the CMFActivity Tool, you can start an activity for each object, then each activity is called one by one.

How to start an activity

You have to call the activate method on the object on which you want to call a method or script. Let's say you want to start an activity in order to set the title, you can do like this :

  my_object.activate().setTitle('foobar')

Like this, the title will not be set immediately, but it will be set some time after by the activity tool.

All arguments and parameters of the method or the script must be pickleable, so you can use tuples, strings, dictionaries, integers... but it is not possible to use objects from the ZODB (in that case you can just pass object path and retrieve them later using restrictedTraverse).

NB: This thread has some more detailed explanations, which should be integrated in this page.

Manage the list of activities

In the zope management interface of your site, you can select the object with the id portal_activities. Then you can select the activity tab and you will get the full list of activities.

Usually, activities are executed automatically. If there is many activities, it may require some time.

On each activity, it is possible to select invoke in order to execute the activity right now, or you may select cancel if you do not want to run the activity.

You will see several columns :

The Processing Node

The Processing Node can have many different values :

If the Processing Node is more than 0, this is the number of the node where the message will be executed. If you don't run a cluster, then the number will be always 1, because there is only one node. If you are running a cluster with many Zope servers, then the processing_node can be as big as the total number of nodes where you are running activities.

How to define an ordered execution of activities

It is often required to start an activity when another one (or many ones) will be finished. You can use several kinds of parameters for that.

The first one is after_method_id, it allows to start an activity after another only by giving the name of the previous method.

Let's say you want to define the description after the title, you can do like this:

  my_object.activate(after_method_id='setTitle').setDescription('aa')

List of possible control mechanisms

How to control the behavior of the activity queue

CMFActivity model is pluggable, it's possible to define an activity queue class, and later pass the id of this class as activity= keyword to activate. CMFActivity comes with the following activity queues:

SQLDict (the default)

SQLDict uses SQL as a backend, so this ensures that activities will be executed even if the server is restarted, and allows to distributes activities in a ZEO cluster. It guarantees that if a message for a method M on an object O will be executed only once at a time (think that (M, O) are the keys of a dictionary and the message is the value). If there is a tuple (M, O) in the queue, calling O.activate().M() will not add a new message, when this message will be processed, calling O.activate().M() will add the message.

SQLQueue

Like SQLDict, SQLQueue uses a SQL backend, but all messages are executed, as a result, if a message M on an object O is n times in the queue, then M will be called n times

RAMDict

It haves the same semantics as SQLDict for duplicate messages, but uses a RAM storage, so it is not peristent across server restarts and is not shared on the cluster, but can be faster.

RAMQueue

Mixes RAMDict and SQLQueue

Usage of activate_kw

/!\ Note: activity_kw is a way to pass parameters to the activate method which is invoked internally from another method. That is the only way to pass such parameters at the moment, but the design is flawed. Originally, I (yo) invented this parameter only for reindexObject, and it was good enough in this case, because a single method call happened here at one level. Other developers, however, extended the meaning, and started to propagate activity_kw at multiple levels to multiple methods, and this was wrong, since the scope is too large.

Using such code snippet:

   1 object.activate(tag='my_tag').edit(activate_kw={'tag':'my_tag'})
   2 object.activate(after_tag='my_tag').myMethod()

it is possible to have such scenario:

Below code snippet:

   1 object.activate(tag='my_tag').edit()
   2 object.activate(after_tag='my_tag').myMethod()

will do:

/!\ More discussion are on mailing list.

How to store the result of an activity

It is sometimes interesting to get the result of an activity. There is a special type of object that we can create in the CMFActivity Tool called : Active Process. This object can be used in order to store the result of some activities.

You can create an active process like this :

  active_process = context.portal_activities.newActiveProcess()

Then if you want to store the result of an activity on this object, there is the parameter active_process that you can give when you call the activate method.

  my_object.activate( active_process = active_process.getPath()).getTitle()
  my_object2.activate(active_process = active_process.getPath()).getTitle()

Then, with the parameter active_process, the result will be automatically stored on the active process. Then you can get the list of result like this :

  active_process.getResultList()

Here the list of result might be : ['title1','title2']

It might be interesting to start an activity with the result of a previous activity. You can do this by giving the path of the active process to the last activity.

  active_process = context.portal_activities.newActiveProcess()
  my_object.activate(active_process = active_process.getPath()).getTitle()
  my_object2.activate(active_process = active_process.getPath()).getTitle()
  context.activate(after_method_id = 'getTitle'
                  ).Base_processResultList(activate_process = active_process.getPath()
                                          )

Like this, when the script Base_processResultList will be called, then we will be sure that all activities getTitle will be finished and the result will be available on the active process. So the script will be able to use it.

What to do if no messages are executed any more ?

One problem may happen with TimerService, it uses gethostbyname(gethostname()) to guess the address of the current node, so if you change your network configuration, you better check the IP address corresponding to your hostname in /etc/hosts

It might happens that some messages are set to processing but the macking looks like to do nothing (for example the load stay very low).

In this case you might only restart Zope and then everything should working again.

How to monitor activities in a Linux shell

There is script watch_activities available in CMFActivity product bin directory: http://svn.erp5.org/erp5/trunk/products/CMFActivity/bin/watch_activities

The result look like this and will be updated every 5 seconds:

+-------------+------------------------+------------+-----------------+
| count(path) | method_id              | processing | processing_node |
+-------------+------------------------+------------+-----------------+
|           1 | expand                 |          0 |              -2 |
|       16439 | immediateReindexObject |         -1 |              -1 |
|           2 | immediateReindexObject |          0 |              -2 |
|         192 | immediateReindexObject |          0 |               1 |
|         101 | immediateReindexObject |          1 |               1 |
|          45 | vincent_repairUids     |         -1 |              -1 |
+-------------+------------------------+------------+-----------------+

HowToUseTheCmfActivityTool (last edited 2010-06-03 11:30:59 by NicolasDelaby)