Plone, GAE, HTML5

The year started with business as usual with our main customer EEA and with overtime we also managed to redesign Jeeves Information System's multilingual websites, all with Plone. I, Sasha went to the Lovely Snow Sprint where the topic this year was Google App Engine. It was a perfect place to meet friends and learn something new. Together with Jens W. Klein from Bluedynamics and Nick Johnson from Google we created TweetEngine for sharing twitter accounts. After the Snow Sprint I was very inspired to do more projects so we decided to take 2 days a month to do none customer work and Per Thulin really let his creativity flow. We put all our projects on http://labs.valentinewebsystems.se/ . During summer Frederik Jeppsson joined us and helped us evaluate  python e-commerce systems and migrate lavendeldockor.se from EasyShop to LFS.

Father and we won the tender

After summer the changes started. I became a father. Per Thulin decided to start his own company so I decided to leave the office and move to the open space at Malmö Arena at St.Gertrud. Together with Eau de Web in Romania we won the tender at EEA! But after more then 6 years I decided that it was time for me to do some new projects so for the new work at EEA I found a replacement for me.

The launch at EU parliament

During the year our biggest project was SOER 2010 and a subproject SENSE for the European state of environment report. But before the launch of SOER 2010 at EU parliament in Brussels I also went to Plone Conference 2010. At the Plone Conference 2010 I gave two talks,   External E-commerce and Plone playing along and Managing multilingual sites.

New year, new opportunities

Now few days left of the year I must say it has been a good year, a lot of work, met interesting people in the SENSE project and I am looking forward to the new year with a lot of new interesting projects and opportunities. 

Merry Christmas and Happy New Year from a cold and snowy Sweden!

References

TweetEngine

Lavendeldockor.se

SOER 2010 - European Environment Agency

Jeeves

SOER 2010

"SOER 2010 provides a set of assessments of the current state of Europe's environment, its likely future state, what is being done and what could be done to improve it, how global megatrends might affect future trends… and more."

Plone powers the environment

In this project Plone plays a big role since it is EEA's main CMS for the public website. The SOER report is today a sub site with content from EU  and neighborhood countries. Two third of the countries uploaded content manually through Plone CMS and 1/3 through RDF feeds as part of the SENSE project. The 13 SENSE countries used their own CMSs where Plone was one of them in at least 3 countries. Most of them used the eea.soer/eea.rdfmarshaller products developed by us for EEA but Italy also had own products for their Plone site.

From static paper to full traceability

The goal of the SENSE project was to give countries more control over their content and possibility to update it continuously and give the readers full traceability to data sources. This has not fully been achieved but it is on it good way and discussions have started in many countries how to make the underlying data available to the public.

Now when the report is launched the discussions are already going on how to continue the work with many other 'sharing information' projects like INSPIRE and SEIS. We hope that more RDF and Linkeddata projects will be the outcome of this.

RDF and Linkeddata in Plone

For this project two products were developed, eea.rdfmarshaller which is a generic RDF marshaller (export only as of today) for any Archetype content. You just add /@@rdf to the url or use a RDF browser which sends correct Accept headers and you will get the content as RDF. With this approach we have enabled any Plone site to speak RDF and we can easy create connections between Linkeddata RDF storages or Plone sites powered by eea.rdfmarshaller.

RDF marshaller allows custom adapters which were utilized in eea.soer to generate specific RDF output in the SOER/SENSE project.

Make websites and make difference

I, Sasha Vincic have been working with EEA for over 6 years and recently I decided that it is time for me and my company to find new projects. The environment, economy and health are important topics and listening to Prof. Jacqueline McGlade, Executive Director of the EEA, Janez Potočnik, European Commissioner for the Environment and specially Jo Leinen, Chairman of the Committee on the Environment, Public Health and Food Safety i was inspired and proud of have been part of the web team at EEA for so many years. Last time I was this inspired this much was at the Eben Moglen's keynote at Plone conference 2006 where he talked about how open source really can make difference in the world. Time to make some difference.

The European environment – state and outlook 2010 (powered by Plone)

The clear sign of mature product is when you realize it can't do everything and it is much better to use the best tool for the job then reinvent and try to shoehorn everything into Plone.

Restful search

Calvin Hendryx-Parker from Six Feet Up told us how they have integrated Solr into Plone. In his talk Enterprise Search in Plone using Solr he explains how alm.solrindex plays very well with ZCatalog and you don't have to do any changes to your code, just install it as a normal ZCatalog index.

David Glick from Groundwire went through the technical aspects of how to use web services from Zope/Plone. In his talk The Art of Integrating Plone with web services he mentions REST, XML-RPC, and SOAP. The interesting thing is that Zope has had XML-RPC support over 10 years! You have been able to provide and consume web services with Zope and Plone for many years.

Diazo the theming service

Me, Sasha Vinčić did a presentation External E-commerce and Plone playing along where I presented how you can use external e-commerce and I also mentioned that you can look at the theme as a service. With the tools Deliverance and Diazo(former XDV) we are able to see it as a service. As with many services we get the benefits of being able to exchange it and reuse it across other services. Easier and faster Plone theming with Deliverance and xdv by Nate Aune, Jazkarta and Theming with XDV by Laurence Rowe and Diazo will be the default theming in Plone 5 indicates that this is the way to go, see themes a service.

How do you use web services with Plone or provide them from Plone/Zope?

Plone CMS or framework

Plone is a CMS not a framework and trying to shoehorn products as add ons is not always the best option. E-commerce are big and complex systems and not simple plugins.

Today we have EasyShop GetPaid and PCommerce which are installable into Plone. None of them is perfect and almost all of them are abandoned or have very little development. When they are installed they are hard to remove because your Plone CMS has become a product inventory, a order management and customer management system. Keeping this out of Plone will make your Plone experience much better.

Service Oriented Architecture

SOA has been around for a while and we use web services like Google Maps everyday. When you create or need new functions for your website think API - web service. This way you can use the best tool for the job or an existing tool like e-commerce system.

Benefits beside the best tool for the job is that your Plone site is easier to upgrade because the CMS is not polluted with a complex system. You also have the flexibility to exchange services when needed or choose to host some of them in the cloud.

E-commerce and theme as a service

When we looked for the e-commerce system we had the requirements that it should be python, it should have a good open source license and be actively developed. We could only find two python e-commerce systems, Lightning Fast Shop - LFS and Satchmo both based on Django. In our case study, Lavendeldockor we choose LFS because it was clean, small and fast. We also implemented the theme with XDV so we could reuse it for both Plone and LFS.

Plone conference 2010 presentation and further information

For more information you can watch the Plone conference presentation about External E-commerce and Plone play along or contact Sasha Vinčić at +46708 840 660 or online form.

Plone's powerful workflow system let you control security of each object in each state. With time-base publishing you want a new document to appear first when the effective date has been reached. In some cases you need to keep the document safe and protected from public before the effective date is reached. With Plone standard workflow the document that is in published state has already permissions which allows public to access it if they know the url even if it should be published in the future. In most cases this is enough since Plone will hide the document from searches and listings but in some cases it would be a disaster if someone accessed a document before the intended time, i.e stock market information.

Add an automatic transition to your workflow that is triggered if the publish date is in the future. Make it go from 'published' state to  "Published (future)" which should have the same permissions as the Private state. Then use the zope built in cron like system to call a simple script that finds all content in 'Publshed (future)' state with publish date before now and publishes them.

This way you will avoid security by obscurity, i.e having accessible content laying around on your websites even though it should be protected.

Need help with Plone?

Contact Sasha Vinčić at +46708 840 660 or online form.

Do you want to know more about Plone?

.

License: ZPL

Technologies: Python 2.4, Plone 3.x (Plone 4 support in beta)

Extendability: Skin layers, portlets

Listed sites/users: 16

Can be run in Google App Engine: No (Plone/Zope is not compatible)

Complete solution, add-on for Plone.

Features: Donations, pay-to-publish, any content "buyable", discounts, UPS shipping (with real-time shipping charge calculation), good documentation, tests(!), getpaid products are available in a multitude of languages and all products are tagged with translatable text-ids.

EasyShop

License: GPL

Familiar technologies: Python 2.4, Plone 3.2.5

Extendability: Skin layers, portlets

Listed sites/users: A handful (most being migrated to LFS?)

Can be run in Google App Engine: No (Plone/Zope is not compatible)

Complete solution, add-on for Plone.

Lightning Fast Shop

License: BSD

Technologies: Python 2.6, jQuery, jQuery UI, TinyMCE, Django

Extendability: Product templates, portlets

Listed sites/users: 11 http://www.getlfs.com/live-shops

Can be run in Google App Engine: No (Python 2.6 can't be run)

Easy to set up, clean management interface, functional shop out of the box and well thought out design. Snappy performance!

Satchmo

License: BSD

Familiar technologies: Python, Django 1.2.1

Extendability: Very high

Listed sites/users: At least 67

Can be run in Google App Engine: No

Satchmo is a complete web shop like the rest. The admin interface works fine, is snappy but is a little bit cluttered and asks tons of questions.

Conclusion

All the tested solutions are functional out of the box. The two Plone based solutions have an advantage in having a tool chain that is familiar to us, but the Django based ones have better performance. LFS stands out because it also looks good out of the box

What is your experience?

Please send us comments and tell us your ecommerce experience. Which system is your favorite?

Links

• http://www.plonegetpaid.com/
• http://www.geteasyshop.com/
• http://wiki.github.com/ned14/Easyshop/starttoendtutorial
• http://bitbucket.org/diefenbach/lfs-buildout-development
• http://www.getlfs.com/week-lfs-42
• http://www.satchmoproject.com/

For a long time, the HTML web standard has been standing still and innovation has mostly happened within proprietary plug-ins such as Flash. Today though, the future of the web is seen within the next generation HTML5 and CSS3 standards.

HTML5 and CSS3 is the new industry standard that leverages:

• Drag & drop
• Video
• Audio
• 2D and 3D graphics and visualizations
  
• Transitions
• Games

To keep up with all of this, Valentine Labs was formed by our front-end expert Per Thulin. Per comes from a creative game development background with lots of experience in multimedia, graphics and visualization. In just a few weeks we have come up with some absolutely amazing projects:

• Valentine Notes - A CSS3 technology demo that mimics the UI and transitions of Apples iPhone Notes application.
• ScriptMate - An application for theater actors to create and rehearse scripts together.
• Timeline Project - Helps people reach their daily goals.
• Tweet Arena - A CCS (content combining system) where the content is hosted on social media sites and web services.
  

Is your web site ready for the future?

Contact Sasha Vinčić at +46708 840 660 or online form.

Try our web applications

 More on HTML5:

• http://slides.html5rocks.com/#slide1
• http://en.wikipedia.org/wiki/HTML5

For the leading supplier of ERP systems - Jeeves Information Systems, only the leading CMS, Plone, is good enough. Valentine Web Systems have delivered two websites. The first is targeted at investors where it is important that press-releases are published on time and not available to the public before they are published. The other website is translated into 11 languages where each language represents the countries where Jeeves has got local partners. Each partner has a login and can maintain their content themselves. The front page content is fully controlled by the editors.

"In May 2010 we launched a new website which we have worked with very intensively since the beginning of this year. With the new website, we now have simple management of press releases, more multimedia plus a powerful and easy to use system for multilingual management for the local partner sites. During the whole development process cooperated closely with Valentine Web Systems who showed strong commitment and have been very helpful. Sasha and Per at Valentine have got enormous knowledge, are creative and will not give up until they find the best solution. They have the ability to combine their technical know-how with a good understanding of customer needs.

Thank you for a great job!"
Cecilia Kullberg
Jeeves Information Systems

Are you looking for a commited provider?

Contact Sasha Vinčić at +46708 840 660 or online form.

Do you want to know more about Plone?

Recently we upgraded our customers website for large data handling. The main development of the product eea.datatservice was done by our colleagues at Eau de Web in Romania, and our part was to prepare Plone 2.5 to handle large data.

Back to the future - blobs in Plone

The first solution for external storage was iw.fss, but due the cluster setup on our production server, we went with the blob approach because it is more future proof since it is part of the upcoming Plone 4 and doesn't require any shared read/write storage for all instances compared to iw.fss.

We tried out the existing branch of blob support for Plone 2.5 but it was old and lacked functionality, so we had to cut a new branch to backport the latest plone.app.blob. While Andreas Z was working on it we where back-porting the code more or less day by day :) Today the blob support for Plone 2.5 allows files and images to be stored outside ZODB in blobs.

In eea.dataservice we have custom content types with large files and images where some are 18000x16000px! All these maps and figures are converted in to number of different formats. All the images are scaled in different sizes and everything is stored in blobs. To store scales in blobs we backported plone.app.imaging which is used in Plone 4. 

Fast but small servers - can't cache everything

In our cluster that is running on blade servers with very little local storage, we have to consider what to cache and where. The large storage on these servers is mounted from SAN, which is a fast and secure storage. The blob storage is mounted on the machine that runs the ZEO and additional to that we have smaller blobcaches for the instances on each machine, 25GB/machine. Since we can't fit the whole blobstorage in cache we clean it manually with a cron. In the newer ZODB3 and Plone there is configuration for automatic cleanup but we can't use it with Plone 2.5. This manual cleaning outside Plone has raised an issue where an instance expects the blob in the cache but it's not there. This happens if the object referring to the blob is in the object cache of the instance. If it is not the problem never arise since the blob is reloaded from ZEO. We are now testing different configurations for the object cache size before we try to catch those exeptions and try to reload the blob from ZEO instead.

Open source

All code used for this website is open sourced and available in Plone or collective repositories and Eionet SVN for i.e eea.dataservice. 

Plone grew with 50GB of data and 20% more traffic

www.eea.europa.eu the Plone 2.5 site is now getting 20% more traffic and with 50GB more data than before the migration. Next step will be to migrate the multimedia and other large content to blobs which will probably free almost 10GB from the Data.fs.

Want to know more?

Please contact Sasha Vinčić for more information. Contact details on the right.

The grid for the open space was filled up with over 40 talks in just of couple minutes, that is more then the scheduled ones.

Valentine Web Systems team was represented by Per Thulin and me, Sasha Vinčić. I gave a talk about 'Managing multilingual sites' which I hope gave some enlightenment about use cases and pitfalls with multilingual sites. You can see the slides below and recording at ustream.tv.

Plone developers and integrators are as always busy working on interesting solutions. I understand why Microsoft's Jeff Tepper thought Plone is Sharepoints open source competitor. Plone may not be the most googled CMS but it is definitly the most secure and the biggest open source competitor to Sharepoint, Websphere and Documentum. Kees Hink and Kim Chee Leong told us how they hardened Plone to military strength. It was audited by Ernst & Young & Pine Digital Security granted certifiedsecure.eu certificate!

Valentine Web Systems provides multilingual and very secure portals built with Plone. If you want to know more, please contact us.

Tjärö is a small and idyllic island in the Karlshamn archipelago in the Swedish southern region of Blekinge. The island is own by the Swedish Tourism Foundation and is for example equipped with a youth hostel, a camping ground, a restaurant and conference facilities. In other words: Most of the things that are needed for an IT sprint with a big gang of happy geeks.

This was how the stage looked like when Tomas Wennström invited some of his friends to a meeting throughout August 21-23, where he expected 10 people at the most to show up. Well, he ended up facing a giant looking interest from people from everywhere to join the party. 286 persons from all the Nordic countries took the boat from Karlshamn to Tjärö to attend Sweden Social Web Camp 2009, as the sprint was called. Sasha Vinčić from Valentine Web Systems was one of them. Valentine Web Systems was, by the way, one of the sponsors of the event.

The main idea behind Sweden Social Web Camp (SSWC) was to give hard working internet geeks from different places a chance to join up and enjoy some nice time together. Just to get a voice and a face on some of them who nowadays turn up on Twitter and other social forums on internet was gold worth. The fact that people from Norway, Finland and Denmark also showed up gave the event a border crossing touch.

The sprint on Tjärö was not only about party and glamour but most of all a brainstorming society where everyone could share experiences and know how about Social Web. The scheduled sessions where the web people could share this valuable stuff with each other were very crowded. Dinner, brunch and peaceful moments by the camp fire were other opportunities for deep, thoughtful and friendly conversations

The aftertaste for every attendant of SSWC has been like enjoying a tasty candy bar. A sprint like this creates everlasting bounds of friendship and the words of joy and appreciation has been numerous after SSWC. Everyone agrees about the thought that this is far from a one hit wonder.

Says Tomas Wennström, the man behind SSWC: “I think we made internet history on Tjärö island.” You can´t describe it better than that.

Me, Sasha Vinčić(twitter), and Per Thulin(twitter) had an amazing time. This was Per's first sprint and he says it was everything he expected from what I have told him :) Over 20 people met up in Bristol to improve Plone CMS and have fun during the Balloon fiesta.

Performance testing and benchmarking

I worked with Tom Lazar, Alan Hoey and Matthew Wilkes on plone performance testing and benchmarking with funkload.  We continued on the collective.funkload by Ross Patterson and added a recipe collective.recipe.funkload wich is a convinient recipe to add funkload to your buildout. With funkload you can then run tests, benchmark and create reports from the results to see how the performance changes. Andreas Zeidler showed me mr.ripley which takes a normal log in combined form and replays it against the domain url you tell it too. This can be used to test new configurations of same buildout or to warm up a site after restart so objects get cached. I then realized this can be used to create funkload tests and use them for benchmarking a site with real life traffic patterns. I added the proxy functionality to mr.ripley (not in svn as of writing) and run the replay through funkload recorder proxy and wola we have tests from real traffic.   

TinyMCE replacing Kupu in Plone

Per worked with Ben Mason, James Guest and Owen Curtis-Quick on the TinyMCE track in order to improve the integration with Plone, especially with regard to inserting images. This involved both working out ways to provide available image size dimensions that have been configured to the specific content type, and making the user interface more consistent with the rest of Plone. TinyMCE is the content editor that will be used in future versions of Plone, and a big reason for that is that it produces clean and valid markup. Therefore, another important fix that was done was to add some missing attributes on the image tag being produced. Rob, the maintainer of Products.TinyMCE happily reports that all this work has been merged into trunk and thus will become part of the next release of Products.TinyMCE!

XHTML validation, transmogrifier, 64bit acquisition and much more

I just want to mention some other topics that were worked on during the sprint. XHTML validation through automatic testing, you can write tests to validate your markup. This is important for tools like Deliverance.

Hanno Schlichting worked on some deep stuff like fixing C code for acquisition on 64 bits machines. There was bugfixes for LinguaPlone and release of transmogrifier which is a powerful transformer. Formgen, override skin templates/scripts with view... and probably some things I have missed.

The first session I attended was FluidDB by Terry Jones. Terry described it as "database with heart of wiki" and its a description I can support from what I have learned. It is a database where objects don't have owners so anyone can add and change metadata(attributes/tags) on them. But the tags have owners and therefore permissions. The big benefit from this approach is that you can add tags or metadata about the object to the object it self instead of creating your own database. Some good examples are all those Twitter apps out there which add or calculate information about the twitter user or the tweets and then store them in their own database. If Twitter used FluidDB everyone could store it's own extra metadata where the original objects where. Now let see if all the promises will be fulfilled when FluidDB is released in a month or so. I really hope they will because I heard Terry sold his apartment to invest in this, that is what I call an entrepreneur! There were two more sessions related to database, one about MongoDB and one about CouchDB from Mozilla. The later uses the language Erlang to get the concurrency speed and it is a replicable and scalable database.

Javascript is THE language for web 2.0 applications and it's much simpler to develop today with all the AJAX libraries out there but still hard to keep up with all the different browsers and it's bugs. That is specially true if you don't use it daily, like a python programmer like me. I was pleasantly suprised to see Pyjamas come this long way. Today it's possible to develop your user interface in Python and have it compiled to javascript that runs in the browser. This is definitely something we will try out because it also enables you to create desktop applications. Luke Leighton who presented "The Zen of Pyjamas" even thought the HTML UI version was much nicer and had better quality then any widget library like PyQ4, PyGTK, wxPython. The main reason for this was that there had gone far more development hours into the web components.

Fulltext search is one powerfull feature in Plone that we use in our daily job. But on large websites it's good to separate the indexing and searching to separate server or service. The "Search with Python and Xapian" was a nice session by Richard Boulton where I saw how to use Python to index and search in Xapian as an external web service.  I also saw Open Object in action from the Open ERP project and I was impressed how simple it is to create enterprise applications with it. Walking back to hotel I thought "What is the difference between Open ERP and SAP?"  SAP has limitations on what it can do and no limitations on the cost, Open ERP has no limitations on what it can do because it's open source and the price well you decide. If you google for Open ERP and SAP this was top hit for me :)  http://fptiny.blogspot.com/2008/10/when-sap-meet-open-erp.html

Valentine Web Systems will be represented at EuroPython 2009 by company founder Sasha Vinčić who will attend the conference this year. Sasha Vinčić is looking very much forward to the EuroPython 2009 conference to get to learn more about what is going on in the wonderful world of Python. Cloud computing and web are interesting subjects.

Meeting old friends who also attend EuroPython 2009 is another highlight. Like Matt Hamilton from Netsight Internet Solutions who is giving the ”Lipstick on a Pig” talk about using Deliverance to skin third party system or Bernd Dorn from Lovely Systems with the talk ”Real World App Engine Projects” about how to get the best out of App Engine, and related technologies such as Bigtable, Amazon S3, EC2 and Cloudfront. There will of course be many more familiar faces to meet in the crowd, be sure about that.

There are a big number of talks that look really interesting to a happy Python fanatic from Sweden, such as Reliable Python Web Sites, Semantic Apps With CubicWeb, Tapping Into The Web Of Data, The Zen Of Pyjamas and Castles In The Cloud.

The week in Birmingham will be a joyride beyond all borders. See you there!