Indexing in Plone got twice as fast
First day of Perfomance sprint in Copenhagen has passed and it really feels like we all have gotten productive from the start. 10 of us are seating at Symbion where Headnet is located and we have two remote sprinters.
Me and and Matt Hamilton started to work on improving the speed of cataloging while you move/rename a tree of objects. Matt had already an idea about using md5 hash on ZCTextIndex which in a quick test gave 30% speed improvement when you edit a document but don't change the body. But when we renamed objects we didn't get this improvement because the object got unindexed and then indexed on the new path so we tried do some magic in ObjectWillBeMovedEvent which would move the index in catalog for this object to the new path instead of unindexing and then get reindexed on ObjectMovedEvent instead of clean index. This way we could benefit from our speed improvement with md5 since the body isn't changed.
This was quickly done and the profiling after this indicated that object_provides indexing is taking way too long. We found out that the calls to zope.component.interface.interfaceToName was to be blamed for this. We tried some simple caching on it with plone.memoize and yeah! We slashed the indexing time by half on rename on a site with 300 random content objects! We let Martijn Pieters take a look at the method and why it was so slow just for an interface name. He found out that Plone really shouldn't use this for object_provides since it was there for a specific use case (I hope Martijn can comment on which) .
Martijn commited the fix, so current plone trunk has an indexing that is twice as fast as yesterday. This commit does not include the magic on ObjectWillBeMovedEvent and md5 so expect more speedups :)
NB! This fix is for trunk (plone 3.x) but me and others have the object_provides index in older Plones so please check the diff (see also) and fix your indexes to get some speed.
Thanks to Nate at Jazkarta for sponsoring, we got a nice sushi delux meal for dinner yesterday.