Importing contacts from OSX Addressbook to Emacs BBDB

A major aspect of using Emacs is that it is more than a text editor – it is more of an application platform – some would call it an Operating System – that allows a host of applications to run within the same context and session – and vastly enrich the user experience. This is in fact a primary reason why many Emacs users spend their entire computer interaction via Emacs.

This unique aspect of Emacs is made possible by the availability of Emacs Lisp as a first class scripting/programming interface for the editor – and has made possible application packages such as W3 (a web browser), Gnus (the NNTP and mail reader), Eshell (a lisp based shell), Wanderlust (a very capable email client), and the Insidious Big Brother Database – aka BBDB – which is a contact manager and the subject of this post.

(BTW, the list of application is only a very small fraction of what is actually available on Emacs – see www.emacswiki.org for a ton of other packages).

As EmacsWiki points out, BBDB is:

“A complete address book for nearly every email and news client written for Emacs, and its functionality supports (by default) making entries in the address book automatically when reading email or posts by people, and easy lookup of addresses while composing messages.”

In essence, it is an always available address book that works with other packages such as mail, Gnus and org-mode to provide a easy to access repository of contact information. The repository itself is a plain text lisp file which by default is stored as ~/.bbdb.

However, as with any contact manager, the problems are usually with:

  1. Initial data import, and
  2. Synchronization with other repositories and devices

For my platform (Apple OSX), the first problem above translates to data import from my primary OS level address book – the AddressBook.app. The second problem is not much of an issue for me since I use AddressBook.app as the primary contact list elsewhere and sync my devices to that. For keeping BBDB in sync, I have a cron job to automate the scripts below.

The contacts import to BBDB requires a data conversion of the address from custom format in which AddressBook.app stores the contacts to the BBDB lisp format. The steps required are as follows:

  1. Extract the records from AddressBook.app
  2. Parse and reformat the relevant fields that BBDB requires, and
  3. Output the reformatted records in the BBDB lisp format

Luckily, for step # 1, an open source command line utility already exists – appropriately called contacts. It is available from http://gnufoo.org/contacts/ and is a svelte 194KB download. This utility allows querying of the OSX Address book and extraction of specific fields and records. The output is a simple CSV text file.

For step # 2 and #3, additional scripting is required. I have written a simple Ruby script called contacts2bbdb.rb to run the contacts utility and output the BBDB repository file:

#!/usr/bin/env ruby -w
#
# Filename: contacts2bbdb.rb
# Description: Converts addresses from OSX Addressbook.app to BBDB format
# Author: Anupam Sengupta (anupamsg ... A-T ... G - M - A - I - L )
# Maintainer: Anupam Sengupta
#
# (c) 2007-2016 Anupam Sengupta.
#
# Created: Fri May 25 15:53:22 2007
# Version: 1.0
# Last-Updated:
#           By: Anupam Sengupta
#     Update #: 733
# URL: https://slashusr.wordpress.com
# Keywords: `BBDB’, ‘OSX’, ‘contacts’, ‘convert’
# Compatibility: GNU Emacs 21 and above
#
#--------------------------------------------------------------------
#
# Commentary:
#
#  Converts the addresses and contacts from Apple OSX’s system addressbook
#  format (Addressbook.app) to Emacs’ BBDB format.
#
#  Requires the following additional software:
#
#  1. Ruby V1.8 and above (http://www.ruby-lang.org/)
#  2. Big Brother Database (BBDB) package for Emacs (http://bbdb.sourceforge.net/)
#  3. The ‘contacts’ program to read Addressbook’s contacts
#     (http://gnufoo.org/contacts/)
#
# Usage:
#
#  1. Install Ruby, BBDB and contacts, if not already present
#  2. Backup the OSX Address book (export the addresses as addressbook archive)
#  3. Run this Ruby script to generate the converted records in BBDB format in the STDOUT
#  4. Save the STDOUT output of this script to bbdb.new
#
#     $ ruby contacts2bbdb.rb > bbdb.new
#
#  5. Replace your .bbdb file with bbdb.new
#
#--------------------------------------------------------------------
#
# Change log:
#
#
#--------------------------------------------------------------------
#
# License: GNU GPL V2.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; see the file COPYING.  If not, write to
# the Free Software Foundation, Inc., 51 Franklin Street, Fifth
# Floor, Boston, MA 02110-1301, USA.
#--------------------------------------------------------------------
#
Delim = “\t”                    # Default delimiter
ContactsProg = ‘/usr/bin/env contacts’ # The contacts program
#
ContactsProgParams = ‘-H -l’
#
# A map between the LDIF field names and the actual field name
ContactFields = {
:fn => ‘firstName’,
:ln => ‘lastName’,
:c  => ‘company’,
:nn => ‘nickName’,
:he => ‘homeEmail’,
:we => ‘workEmail’,
: oe => ‘otherEmail’,
:hp => ‘Home’,
:mp => ‘Mobile’,
:Mp => ‘Main’,
:wp => ‘Work’,
}
#
ContactFormatOpts = ContactFields.keys #  Options to pass to the contacts program
ContactsProgFormat =  “‘%” + ContactFormatOpts.inject { |s, f| s.to_s + “#{Delim}%#{f.to_s}” } + “‘“
ContactsFullExec = “#{ContactsProg} #{ContactsProgParams} -f #{ContactsProgFormat}”
#
output = `#{ContactsFullExec}`.split(/\n/) # Read the output of contacts program
#
# Start parsing the contacts output
records = output.map do |line|
record = Hash.new(nil)
line.chomp.split(Delim).each_with_index { |f, i| record[ContactFields[ContactFormatOpts[i]]] = f.strip unless f =~ /^\s*$/ }
record
end
#
# Start outputing the details to STDOUT
puts <<END
;; -*-coding: utf-8-emacs;-*-
;;; file-version: 6
END
#
for r in records do
r[‘nickName’] = nil           # No need for the nick names.
outs = %w{ firstName lastName nickName company }.inject(“[“) { |s, f| s + (r[f] ? “\”#{r[f]}\” “ : “nil “) }
outs = %w{ Home Main Mobile Work}.inject(outs + “(“) { |s, f| r[f] ? s + “[\”#{f}\” \”#{r[f].strip}\”] “ : s } + “) “
outs = %w{ homeEmail workEmail otherEmail }.inject(outs + “ nil (“) { |s, f| r[f] ? s + “\”#{r[f]}\” “ : s } + “) “
outs += “((creation-date . \”2009-02-08\”) (timestamp . \”2009-02-08\”)) nil]”
puts outs
end
# End of contacts2bbdb.rb script

A companion shell script to automate the execution of the Ruby program and also backup the previous BBDB file is also provided. Before usage, please edit the BBDB folder and file names:

#!/bin/sh
#
# Filename: contacts2bbdb.sh
# Description: Converts addresses from OSX Addressbook.app to BBDB format
# Author: Anupam Sengupta (anupamsg@gmail.com)
# Maintainer: Anupam Sengupta
#
# (c) 2007-2016 Anupam Sengupta.
#
# Created: Fri May 25 15:53:22 2007
# Version: 1.0
# Last-Updated:
#           By: Anupam Sengupta
#     Update #: 740
# URL: https://slashusr.wordpress.com
# Keywords: `BBDB’, ‘OSX’, ‘contacts’, ‘convert’
# Compatibility: GNU Emacs 21 and above
#
#--------------------------------------------------------------------
#
# Commentary:
#
# Runs the export of OS X Address Book contacts to the Emacs BBDB file.
#
# Requires:
#   1. The contacts2bbdb.rb Ruby script
#      (https://slashusr.wordpress.com/)
#   2. The Ruby interpreter (Version 1.8 and above)
#      (http://www.ruby-lang.org/)
#   3. The Big Brother Database (BBDB) Emacs Package
#      (http://bbdb.sourceforge.net/)
#   4. The contacts command line program to extract the records
#      (http://gnufoo.org/contacts/)
#
# Usage:
#   1. Backup the Addressbook.app’s addresses
#   2. Install BBDB and ‘contacts’, if not already present
#   3. Edit the BBDBDIR and RUBYPROG variables below
#   4. Open Terminal.app and run this script
#   5. The script will also create a backup of the previous bbdb file
#
#
#
# TODO: Edit these variables to match the file locations
BBDBDIR=$HOME/.emacs.d/data     # Edit this to match location of the bbdb file
BBDBFILE=bbdb                   # Edit this to match your bbdb file name
RUBYPROG=$HOME/bin/contacts2bbdb.rb # Name and loction of the Ruby Script
#
#######################################  Do not edit beyond this point !
#
RUBYEXEC=`which ruby`
#
if [ -x ${RUBYEXEC} ]; then
cp ${BBDBDIR}/$BBDBFILE ${BBDBDIR}/bbdb.bak
${RUBYEXEC} ${RUBYPROG} > $BBDBDIR/$BBDBFILE
fi
echo “BBDB Export and conversion completed.”
#
# End of contacts2bbdb.sh

Do remember to take a backup of your existing BBDB file as well as a backup of the OSX address-book before running the scripts!

Enjoy!

Update [17th Aug 2009]: Emacs-fu has a nice bbdb tutorial at emacs-fu: managing e-mail addresses with bbdb

Replacing Mail.app with Gmail on OSX

Off late the Mail.app email client on OSX has been acting up on me. It often stalls or displays the spinning beach ball of death till a forced quit is required. Also, it seems to have display rendering issues when a mail classified incorrectly as junk is moved back to the inbox.

I tend to use the Mail.app as a local email client for three main purposes:

  1. Offline mode for reading my emails when there is no Internet connection available
  2. Easy desktop search for the mails via Spotlight, and
  3. Easy archival of important mail in my project workspaces (either in Devonthink or plain text export)

The other features (TODO and Notes) are nice, but not really useful for me as I use other tools for these items. The mail itself is served out via gmail accounts and IMAP synchronization.

However, Mail.app is one of the few (only?) Apple provided applications that I have a love-hate relationship with. I love the fact that it integrates seamlessly with iCal and Address Book, and the other features such as support for multiple signatures, digital signing and encryption via the GPG plugin and the smart quote while responding are quite good. the threading is also functional.

But the incessant crashes and freeze ups are getting to be more than annoying. I have tried the various cures such as re-synching the entire mailbox, rebuilding the mailbox, trashing preferences, changing the cache setting and a ton of other voodoo. These seem to be temporary solutions however, and the problems come back pretty fast. Apparently I am not alone in my suffering though – Apple’s discussion and support forums are full of unhappy Mail.app users.

So I am now on the hunt for an alternative offline/local client solution. I have already tried Mozilla Thunderbird, and it looks too ugly for my tastes (though an excellent client on the Windows platform). In addition, two more deal-breakers exist right now:

  1. Getting Spotlight to work with Thunderbird requires a third party indexer which seems to be a hack. It looks like an experimental mdimporter plugin does come with version 2.0 but is flacky
  2. Missing integration with the system provided AddressBook.app application (I need to sync the addresses with my Blackberry, and AddressBook.app is a core part of the synchronization workflow)

Another promising solution seems to be using Gmail directly as the primary email client. This requires a few additional steps to enable a seamless offline operation:

  1. Installation of the Google Gears extension for the browser (no support yet for the latest Safari 4.x version) – this enables a copy of the emails to be stored on the local disk and allows access and usage of the gmail interface when offline – note that the first-time sync takes a long time as last 6 months worth of emails are downloaded to your computer – it is pretty peppy from then on
  2. Installation of the Gmail Notifier application which provides notifications on receipt of new emailgooglenotifiersignin.png
  3. Setting up Gmail as the default mail handler from URLs by using an option on the Gmail notifiergooglenotifierpreferences1.png
  4. Using a Site Specific Browser such as Fluid.app or Mozilla Prism and creation of a Gmail SSB application (remember to set a nice icon)fluid1.png
  5. Setting up the Gmail SSB as the default mail application by setting the “Default Mail Reader” option from Mail.app’s general preferencesmail-appgeneralpreferences1.png
  6. Synchronizing the addresses between Addressbook.app and Gmail using iSyncaddressbookpreferences.png
  7. Using Google Desktop for searching the mails on Gmail

Whew! Quite a bit of setup here to do – and it is still not perfect. Lets look at the pros and cons:

Pros:

  1. Can use the excellent Gmail UI everywhere, online or offline – on all platforms. Hurray for Web apps!
  2. Great keyboard shortcuts – much better than Mail.app
  3. Using the SSB allows a smooth integration of the mail experience with rest of the desktop
  4. No more waiting for the local client to download the mails before accessing – it is near instantaneous after the first offline synchronization
  5. No more crashes!

Cons:

  1. It is a hack right now – definitely not a “download and start using” solution
  2. Google Desktop is duplicating search functionality that already exists via Spotlight
  3. Synchronization of the addresses between Gmail and AddressBook is not reliable
  4. No support for multiple signatures (can get around with a auto-typing solution such as Typinator or TextExpander)
  5. Cannot export the emails (not the contacts) from Gmail to the local computer

I intend to use this setup for the next couple of weeks to get a better feel of the system. Will follow up with a post on the findings. Do let me know what you think.

Hurray! NetNewsWire 3.2b13 has removed Advertisements

The latest beta of NetNewsWire (3.2b13) has removed the annoying advertisements that were a major distraction. Also, they have brought back the clipping feature – which is really one of the most useful functions for storing interesting RSS articles for reading later.

However, the resurrected clippings do not sync with Google Reader, which cripples the function. In my usual use case scenario, I used to use the Newsgator online web interface to store long articles for reading later at leisure at home. Looks like this is not going to be possible.

BTW, my quest for finding an alternative RSS reader client on the Mac is still not fulfilled. I had tried out the following RSS readers:

  1. Vienna – A nice and capable OSS reader, but a little too basic and the chrome of the application seems out of place on the Mac. Also, it seems to crash quite often
  2. NewsFire – I had a license from a Macheist deal from 2007, and the latest version is free. However, the functions are really basic

Looks like NetNewsWIre with its latest warts is still the most capable reader client on the Mac … I guess I could use Firefox’s live RSS feeds as well … they do sync nicely via the Foxmarks extension.