Skip to content


Using Yahoo Pipes and Tripolis Dialogue to aggregate news in a weekly newsletter

In my last post about the Vegalicious Weekly newsletter I demonstrated how to use RSS feeds from my blog to populate the content of my newsletter in Tripolis Dialogue. Later I also added news items to the newsletter and wanted to combine news from multiple sites. So I had to find a way to aggregate news items of RSS feeds and limit the results to those of the past week.
The solution came with Yahoo Pipes, a visual tool to aggregate, manipulate, and mashup content from around the web. With Pipes I was able to combine two feeds, filter out only those news items of last week and sort them all by date again. The result is a regular RSS feed, which can be used in Tripolis Dialogue as described previously.

Using Yahoo Pipes and Tripolis Dialogue to aggregate news in a weekly newsletter from Harald Walker on Vimeo.

Posted in email.

Tagged with , , .


From blog to newsletter and back

It is often being said, that software developers should be using their own software, so after many years it was about time for me to get going with my own newsletter. Since I also have a food blog called Vegalicious, the idea was born to send a weekly newsletter with the summary of the posts and comments of the past week.

Managing subscribers
In order to get subscribers for the newsletter I used the new web-forms module in Tripolis Dialogue, which we just released last month. After specifying the database fields and the form details, I only had to add a bit of html with a subscribe form as widget in Wordpress.

Widget and form

For the confirmation mails during the registration process I used my own customized design, similar to the final newsletter design.  

confirm mail

In the last mail we send to confirm the successful registration I included a link to the last sent newsletter. This link appears to be very popular. 89% of the subscribers who opened the confirmation mail also clicked on this link since it was added.

Content syndication
Since the purpose of the newsletter was a weekly digest without causing too much extra work, I wanted to automate the process as much as possible and pull in the content directly from Wordpress. In Tripolis Dialogue we can create articles based on a RSS or ATOM feed or include any kind of XML feed, and process it directly in the newsletter template. As I didn’t want to duplicate our blog posts I choose the second option. The newsletter has 3 sections: an introduction area, a main area for new recipes and reviews and a sidebar for comments.

Handling the comments was the easiest. Wordpress offers comments on blog post as RSS feed, which can be imported directly in the newsletter template:

<#assign commentfeed = xmlImporter("http://www.vegalicious.org/comments/feed/")>

Next I only wanted to display a maximum of 7 comments and in case of a very long comment trim it to 100 characters using standard Freemarker (the templating language we use in Tripolis Dialogue) functions for lists and strings.

<#list commentfeed.rss.channel.item as item>
<a href="${item.link}">${item.title}</a>
<#if item.description?length &gt; 100>
   ${item.description?substring(0,100)} (...)
<#else>
   ${item.description}
</#if>
<#if (item_index+1) = 7><#break></#if>
<hr size="1" noshade="noshade" />
</#list>

Dealing with the recipe and review posts was a bit more challenging. Wordpress can supply a feed per month, but not per week. So I had to use the category feeds, which include the 10 most recent posts (this is a Wordpress setting). In order to fetch only the posts of the last week I turned the number of feed items to display into a variable parameter of an article, which I can easily change per newsletter. The header and feed URL can also be customized in the same way.

feed-article-settings

Next I had to deal with the limitations of the RSS format. Our blog posts usually use one or more photos, an introduction, a list of ingredients, cooking instructions etc. An RSS item has a description with the summary of the post excluding any styling and the content of the post with the full html. For the newsletter I only wanted the text of the introduction and a thumbnail image. The images of the blog post are not available as individual elements in the RSS feed, so I extracted the image urls from the html using a regular expression. As sometimes the blog post already uses the thumbnail version of an image, an extra check was necessary.

<#ftl ns_prefixes={"e":"http://purl.org/rss/1.0/modules/content/"}>
<#assign htmlcontent = item["e:encoded"][0]>
<#assign images = htmlcontent?matches("(<img[^>]*?src=\")(.*?)(.jpg\".*?>)")>
<a href="${item.link}">
<#assign firstImage = images[0]?groups[2]>
<#if firstImage?index_of("150x150") == -1>
   <#assign thumbnail = firstImage + "-150x150.jpg">
<#else>
   <#assign thumbnail = firstImage + ".jpg">
</#if>
<img name="" src="${thumbnail}" border="0" 
width="150" height="150" title="${item.title}" alt="${item.title}" />
</a>

Vegalicious Weekly in GmailFor the trimming the text after the introduction I wrote a small function in Freemarker which looks for the position of certain keywords in the post (”Yield”, “Servings”, “Ingredients”).
The publishing date of a blog post can be formatted with the built-in Freemarker functions for dates: ${item.pubDate?date(”EEE, dd MMM yyyy”)}

The final version of the newsletter uses 3 RSS feeds (more can be added) and has a custom article for an introduction.

The way back
For outgoing links back from the email newsletter to our food blog I used the Google Analytics support which is integrated in Tripolis Dialogue. This way I can relate web-site visitors directly to our weekly newsletter. The newsletter has been sent 3 times now, twice Saturday night and once on Sunday. Below you can see how this resulted in additional visits on the site. Looks like I have to experiment a bit more with the best moment for sending it.

google analytics campaign results

In total this has been an interesting exercise for me. Although I’ve been working on implementations with customers for many years, that is still different from actually being in the customer role yourself. It gave me some new ideas for our product and it would be great if Wordpress would have more flexible content syndication like the blog posts of the last 7 days and images as RSS enclosure element.

Posted in email, freemarker.


Agile traveling

My wife and I just returned from our winter vacation, and this time I noticed some parallels with agile software development that I haven’t been aware of before. We travel by motor home and usually only have the first days and the general route planned. Often there is also a theme or a goal as profane has to be warm and have sunshine.

Each day we decide what to do next based on factors like the weather prediction and interesting places to visit, which we often discover while we are traveling. The nights we spend at special motor home parking lots or campgrounds, depending on availability and our needs like electricity or a hot shower. We usually only spend one day at a place, sometimes two if we have to do the laundry or are tired of driving. This all reminds me a lot of agile software development. When we travel we have a general road-map, one or two day iterations, retrospectives in the evening and the presentation you can soon see on our travel blog and Flickr. If necessary we completely adjust our plans and change course. During this vacation we switched from the Mediterranean to the Atlantic coast because of the weather prediction. A drastic change in course but it also gave us the chance to see new interesting areas where we have not been before.

The flexibility also has a price. The unexpected is often positive and a surprise but sometimes also negative and one has to deal regularly with uncertainty and problems. The alternative would be a fully planned and organized vacation, as so many tour operators are offering them for a lot of money. In the past we have rented an apartment somewhere for the length of the vacation and explored the region around it. Sometimes we returned early as the weather turned bad, or we had seen everything and didn’t feel like staying in the apartment, which was not as nice as our own home.

In spite of some nasty problems on the last tour, I prefer agile traveling, just as I still believe in agile software development. It is not always easy and fun but in the end the result is likely to be better.

Posted in agile, software.


Tripolis scores again in the Emerce 100

The Emerce 100 is a list of companies with the best reputation in the Dutch e-business, based on market research conducted by TNS Nipo.
Last year our company Tripolis Solutions finished 4th in the software category, which was good for number 40 in the total top 100. This year there isn’t a top 100 but instead 20 categories each with a top 5 and now our company scored the first place in the e-mail marketing category. The 4th place in this category went to The People’s Valley, who is a partner of us and uses our e-mail marketing software.

Posted in email.


Filtering mail attachments with maildrop and python

For a new feature of our application we had to filter incoming emails based on header values in attached emails. As I could not find examples on how to filter on attachments directly with maildropfilter, I decided to handle the attachments with an external script. In the solution described below, the script returns a value but instead it could also modify the incoming mail and then one would use the xfilter instruction of maildropfilter. Continued…

Posted in python.

Tagged with , .