In the previous post, we saw that finalizers are difficult to write in the presence of circular references, even when using weak references to hold the finalizer callback. The main issue is that if object A’s finalizer is held by object A, and object A has a circular reference with object B, the finalizer will never get invoked because both objects are removed before the finalizer has a change to be invoked.
There are two strategies to work around this issue: one is to save the finalizers in an external object that does not belong to a cycle with object A, the other is to remove the circular reference with object B. This is essentially what JavaObject1 and JavaObject2 do in the next code snippet:
finalizers = []
class JavaObject1(object):
def __init__(self, id):
self._id = id
self._methods = {}
finalizers.append(weakref.ref(self, lambda wr : inc1()))
def __getattr__(self, name):
if name not in self._methods:
self._methods[name] = JavaMember(name, self)
return self._methods[name]
class JavaObject2(object):
def __init__(self, id):
self._id = id
self._wr = weakref.ref(self, lambda wr : inc2())
def __getattr__(self, name):
return JavaMember(name, self)
class JavaMember(object):
def __init__(self, name, container):
self.name = name
self.container = container
def __call__(self, *args):
j = 0
for i in xrange(1, 10):
j += i
As we can see, the weak reference/finalizer in JavaObject1 is held by the global variable finalizers. JavaObject2 is no longer part of a cycle because it creates a new JavaMember every time a member is requested. The JavaMember still refers to the JavaObject2 instance to prevent the eager garbage collection of the JavaObject2 instance discussed at the beginning of the last post. If we try to test our new implementations, we obtain these results:
def m1():
for j in xrange(0, 100):
java_object = JavaObject1('o' + str(j))
for i in xrange(10000):
java_object.method1()
def m2():
for j in xrange(0, 100):
java_object = JavaObject2('o' + str(j))
for i in xrange(10000):
java_object.method1()
if __name__ == '__main__':
timer(m1,'With JavaObject1: ')
timer(m2,'With JavaObject2: ')
...
With JavaObject1: 1.83600997925
acc1:47
acc2:0
With JavaObject2: 2.38709688187
acc1:47
acc2:100
REMINDER: the float number is the time it took to make 1 million method calls (1oo instances X 10 000 calls), the number beside acc1 indicates how many times a finalizer of a JavaObject1 instance was called (max 100), and acc2 indicates how many times a finalizer of a JavaObject2 instance was called.
First, we can see that JavaObject1 is performing a little better than JavaObject2 because method objects are cached and do not need to be instantiated at every call. Second, we can see that not all finalizers of JavaObject1 instances are called. Why?
Well, the answer lies in the garbage collection strategies used by CPython. Roughly, when there is no circular dependency, CPython uses a reference counting strategy: when the last reference to an object is deleted, the object is immediately garbaged collected (hence the perfect acc2 score for JavaObject2). But when there is a circular dependency, CPython uses a mark and sweep strategy which is only executed once in a while. In other words, the Python interpreter has to stop the program execution, inspect the objects, and clean the objects. The frequency of the garbage collection can be set using the gc module.
In the previous example, because the program did not execute long enough (more precisely, because the number of allocations/deallocations did not reach the gc thresholds), some objects were never collected before the end of the program. If instead we explicitly invoke the garbage collector, we see that now all finalizers are invoked:
if __name__ == '__main__':
timer(m1,'With JavaObject1: ')
timer(m2,'With JavaObject2: ')
print('Running GC')
gc.collect()
print('acc1:' + str(accumulator1))
print('acc2:' + str(accumulator2))
...
With JavaObject1: 1.83600997925
acc1:47
acc2:0
With JavaObject2: 2.38709688187
acc1:47
acc2:100
Running GC
acc1:100
acc2:100
The next post will provide a summary of the lessons learned!
). The release will introduce compatibility with Python 3.0, thanks to the huge effort of a generous contributor! There are also a few fixes to bugs reported by users in this version.
