Using Yahoo Pipes and Tripolis Dialogue to aggregate news in a weekly newsletter from Harald Walker on Vimeo.

Bookmark It

Hide Sites

Managing subscribers
In order to get subscribers for the newsletter I used the new web-forms module in Tripolis Dialogue, which we just released last month. After specifying the database fields and the form details, I only had to add a bit of html with a subscribe form as widget in Wordpress.

For the confirmation mails during the registration process I used my own customized design, similar to the final newsletter design.  

In the last mail we send to confirm the successful registration I included a link to the last sent newsletter. This link appears to be very popular. 89% of the subscribers who opened the confirmation mail also clicked on this link since it was added.

Content syndication
Since the purpose of the newsletter was a weekly digest without causing too much extra work, I wanted to automate the process as much as possible and pull in the content directly from Wordpress. In Tripolis Dialogue we can create articles based on a RSS or ATOM feed or include any kind of XML feed, and process it directly in the newsletter template. As I didn’t want to duplicate our blog posts I choose the second option. The newsletter has 3 sections: an introduction area, a main area for new recipes and reviews and a sidebar for comments.

Handling the comments was the easiest. Wordpress offers comments on blog post as RSS feed, which can be imported directly in the newsletter template:

<#assign commentfeed = xmlImporter("http://www.vegalicious.org/comments/feed/")>

Next I only wanted to display a maximum of 7 comments and in case of a very long comment trim it to 100 characters using standard Freemarker (the templating language we use in Tripolis Dialogue) functions for lists and strings.

<#list commentfeed.rss.channel.item as item>
<a href="${item.link}">${item.title}</a>
<#if item.description?length &gt; 100>
   ${item.description?substring(0,100)} (...)
<#else>
   ${item.description}
</#if>
<#if (item_index+1) = 7><#break></#if>
<hr size="1" noshade="noshade" />
</#list>

Dealing with the recipe and review posts was a bit more challenging. Wordpress can supply a feed per month, but not per week. So I had to use the category feeds, which include the 10 most recent posts (this is a Wordpress setting). In order to fetch only the posts of the last week I turned the number of feed items to display into a variable parameter of an article, which I can easily change per newsletter. The header and feed URL can also be customized in the same way.

Next I had to deal with the limitations of the RSS format. Our blog posts usually use one or more photos, an introduction, a list of ingredients, cooking instructions etc. An RSS item has a description with the summary of the post excluding any styling and the content of the post with the full html. For the newsletter I only wanted the text of the introduction and a thumbnail image. The images of the blog post are not available as individual elements in the RSS feed, so I extracted the image urls from the html using a regular expression. As sometimes the blog post already uses the thumbnail version of an image, an extra check was necessary.

<#ftl ns_prefixes={"e":"http://purl.org/rss/1.0/modules/content/"}>
<#assign htmlcontent = item["e:encoded"][0]>
<#assign images = htmlcontent?matches("(<img[^>]*?src=\")(.*?)(.jpg\".*?>)")>
<a href="${item.link}">
<#assign firstImage = images[0]?groups[2]>
<#if firstImage?index_of("150x150") == -1>
   <#assign thumbnail = firstImage + "-150x150.jpg">
<#else>
   <#assign thumbnail = firstImage + ".jpg">
</#if>
<img name="" src="${thumbnail}" border="0" 
width="150" height="150" title="${item.title}" alt="${item.title}" />
</a>

For the trimming the text after the introduction I wrote a small function in Freemarker which looks for the position of certain keywords in the post (”Yield”, “Servings”, “Ingredients”).
The publishing date of a blog post can be formatted with the built-in Freemarker functions for dates: ${item.pubDate?date(”EEE, dd MMM yyyy”)}

The final version of the newsletter uses 3 RSS feeds (more can be added) and has a custom article for an introduction.

The way back
For outgoing links back from the email newsletter to our food blog I used the Google Analytics support which is integrated in Tripolis Dialogue. This way I can relate web-site visitors directly to our weekly newsletter. The newsletter has been sent 3 times now, twice Saturday night and once on Sunday. Below you can see how this resulted in additional visits on the site. Looks like I have to experiment a bit more with the best moment for sending it.

In total this has been an interesting exercise for me. Although I’ve been working on implementations with customers for many years, that is still different from actually being in the customer role yourself. It gave me some new ideas for our product and it would be great if Wordpress would have more flexible content syndication like the blog posts of the last 7 days and images as RSS enclosure element.

Bookmark It

Hide Sites

Each day we decide what to do next based on factors like the weather prediction and interesting places to visit, which we often discover while we are traveling. The nights we spend at special motor home parking lots or campgrounds, depending on availability and our needs like electricity or a hot shower. We usually only spend one day at a place, sometimes two if we have to do the laundry or are tired of driving. This all reminds me a lot of agile software development. When we travel we have a general road-map, one or two day iterations, retrospectives in the evening and the presentation you can soon see on our travel blog and Flickr. If necessary we completely adjust our plans and change course. During this vacation we switched from the Mediterranean to the Atlantic coast because of the weather prediction. A drastic change in course but it also gave us the chance to see new interesting areas where we have not been before.

The flexibility also has a price. The unexpected is often positive and a surprise but sometimes also negative and one has to deal regularly with uncertainty and problems. The alternative would be a fully planned and organized vacation, as so many tour operators are offering them for a lot of money. In the past we have rented an apartment somewhere for the length of the vacation and explored the region around it. Sometimes we returned early as the weather turned bad, or we had seen everything and didn’t feel like staying in the apartment, which was not as nice as our own home.

In spite of some nasty problems on the last tour, I prefer agile traveling, just as I still believe in agile software development. It is not always easy and fun but in the end the result is likely to be better.

Bookmark It

Hide Sites

The python script reads the email from stdin using the email package and then iterates over all the parts and subparts of the message object tree. In our example we are looking for a header called ‘x-mailer’ and if the value is BigMailer, we return a value to the maildrop filter.

#!/usr/bin/env python
 
import email
import sys
 
msg = email.message_from_file(sys.stdin)
 
for part in msg.walk():
        if 'x-mailer' in part:
                if part['x-mailer'] == 'BigMailer':
                        print "BigMailer"
                        sys.exit(0)

From maildroprc the python script is being executed when there is a match for a certain subject and in case the script returns ‘BigMailer’, the email is being delivered to the mybigmailer mailbox.

if (/^Subject:.*Mail with attachment mail/)
{
    SOURCE=`/opt/csw/bin/python /opt/csw/etc/maildrop/myfilter.py`
    if ($SOURCE eq "BigMailer")
    {
     to /vmail/mybigmailer/
     exit
    }
}

Bookmark It

Hide Sites

We found a working solution in a combination of a custom Ant plugin for Maven, Maven as build tool, shell scripts and Luntbuild. It took us quiet a while to figure it all out and get it right, but now we are deploying multiple versions each night and can also trigger deployments manually. In this post I am going to show how we solved it.

1. Ant Plugin for Maven
We were not able to find existing Maven plugins for the tasks of copying, unpacking and remotely managing applications, so we decided to write our own Ant Plugin for Maven, which allowed us to use tasks like sshexec and scp.
The plugin basically only needs 3 files: an 
ant build script, a mapping document and of course Maven’s POM. The image below shows the Eclipse project for the plugin.

The ant script rollout.build.xml performs tasks like SSH commands and SCP and accepts various parameters, which allow us to use it in a very flexible way. In the code example below, you can see two targets, one for the web applications and one for the daemon processes.

In order to use the ant script, a mapping document has to be created to tell Maven how to use our plugin.

Because we use SSH/SCP, ant-jsch and jsch have to be added as dependencies to the pom.xml of the plugin. Here we also specified the more user-friendly prefix ‘rollout’, so that we can reference our new plugin with rollout:daemon and rollout:webapp. Don’t forget to update settings.xml when using the prefix.

If you want to to execute the script automatically as part of one of the Maven lifecycles, you can add the plugin to the pom.xml of your project as described in the Guide to Developing Ant Plugins. Originally we used the ’site’ lifecycle to create the project sites and deploy the application as well but that slowed the deployment process down too much. Sometime you want to be able to release a new version as fast as possible.

2. Shell scripts for starting/stopping applications
On the test server we are using standard shell scripts to manage the applications. Just as the ant script, these shell scripts accept various parameters, so we can use them for multiple deployments.

3. Luntbuild

In Luntbuild – an open source build automation and management tool – we bring the loose ends together and configure the dynamic parameters for the build and deployment process. The project/scheduler concept of Luntbuild combined with a series of builders make it easy to handle multiple versions of our software.

For the nightly deployment on trunk we use a schedule (screenshot below), which first builds the project (clean install) and then post-build process will setup a database on the test server and deploy all 9 applications.

The builder then uses Maven to execute the deployment goal of our plugin and provides the necessary parameters. As you can see, it is possible to pass parameters from the schedule to the builder and this way it is possible to re-use the builder in many ways. Unfortunately it is not possible to share builders between projects in Luntbuild.

Conclusion

With this workflow we were able to achieve our goal. Deployment is now part of our continuous integration system. Every night each release which is in development is automatically being deployed to our test servers and if necessary we can manually start deployments during the day. All of this saved a lot of time and trouble. It is a very flexible and expandable solutions but the flexibility also added some complexity.

What is missing now are the setup and configuration steps for the test systems and databases. At the moment there are still some manual steps necessary to get a new deployment environment ready.

Bookmark It

Hide Sites

import javax.script.*
 
def takeScreenshot = """\
open location \"http://www.bitwalker.nl\"
delay 5
do shell script \"screencapture screenshot.pdf\"
"""
 
ScriptEngineManager mgr = new ScriptEngineManager()
ScriptEngine engine = mgr.getEngineByName("AppleScript")
engine.eval(takeScreenshot)
engine.eval("say \"Your screenshot is ready\"")

You can run this directly in the GroovyConsole or as regular Groovy script. Don’t forget to set JAVA_HOME to 1.6.

Bookmark It

Hide Sites

Also many talks were quiet full, which according to my experience doesn’t contribute to deep discussions and exchange of thoughts. In large groups typically some and often the same people stand out and dominate the discussion with their opinion, while in a smaller group it is usually more balanced.

So my conclusion is, that also here less is more. A smaller conference might make it more difficult to find sponsors but on the other hand costs would also be much lower. The alternative could be to expand the conference in space and time but then costs would go up. Personally I would rather have a small conference each quarter than one or two large ones per year. 50-60 seems to be a good number of participants for open space conferences.

Bookmark It

Hide Sites

Next, I was looking at the usage of webmail clients like Hotmail and Gmail. About 17% of the email is being read online with a webmail client. Here Windows Live Hotmail is still dominating. Since Google’s Gmail was opened to the general public on April 1, 2004, it has gained a market share of 7.4%. Good old Yahoo Mail finished as 3rd with 6.2%.

Bookmark It

Hide Sites