Instapaper Kindle Two way Sync
Well today might not be the best day for Marco of Instapaper with the introduction of Reading List feature in iOS and Mac OS X by Apple. But I love Instapaper nonetheless. But one of the annoyances I have is that there is no two way sync with the Kindle. Well there is http://goephemera.com/ but it is unmaintained and no longer works for multiple articles.
So in a binge of insomniac coding I decided to write such a sync. Now, I realize that I am not going to be putting a lot of work into maintaing this so take it as is and improve it as you like.
Somethings you will need:
Leave a comment if you need help. But this does assume you have knowledge of the command line.
require 'rubygems' require 'oauth' require 'json' require "sqlite3" # Get your key here. http://www.instapaper.com/main/request_oauth_consumer_token INSTAPAPER_CONSUMER_KEY = 'getyourownkey' INSTAPAPER_CONSUMER_SECRET = 'getyourownkey' INSTAPAPER_USERNAME = 'user@mail.com' INSTAPAPER_PASSWORD = 'password' KINDLEGEN = "~/Dropbox/KindleGen/kindlegen" class Instapaper Url = "http://www.instapaper.com" def initialize(consumer_key, consumer_secret) @consumer_key = consumer_key @consumer_secret = consumer_secret end def authorize(username, password) @consumer = OAuth::Consumer.new(@consumer_key, @consumer_secret, { :site => "https://www.instapaper.com", :access_token_path => "/api/1/oauth/access_token", :http_method => :post }) access_token = @consumer.get_access_token(nil, {}, { :x_auth_username => username, :x_auth_password => password, :x_auth_mode => "client_auth", }) @access_token = OAuth::AccessToken.new(@consumer, access_token.token, access_token.secret) end def request(path, params={}) @access_token.request(:post, "#{Url}#{path}", params) end end $instapaper = Instapaper.new(INSTAPAPER_CONSUMER_KEY, INSTAPAPER_CONSUMER_SECRET) $instapaper.authorize(INSTAPAPER_USERNAME, INSTAPAPER_PASSWORD) $db = SQLite3::Database.new('i2ksync.db') $db.results_as_hash = true # status 1. download, 2. still_there 3. archived $db.execute(%{ create table if not exists bookmarks ( id integer PRIMARY KEY, title text, status text, UNIQUE(id)) }) # Create the path if it doesn't exist path = File.join("/Volumes/Kindle/documents", "_instapaper") Dir.mkdir(path) unless File.exists?(path) num_not_there = 0 still_there_ids = [] still_there = $db.execute("select * from bookmarks where status = 2") still_there.each do |bookmark| unless File.exists?(File.join(path, "#{bookmark['title']}.mobi")) $instapaper.request("/api/1/bookmarks/archive?bookmark_id=#{bookmark['id']}") $db.execute("update bookmarks set status = 3 where id = #{bookmark['id']}") num_not_there += 1 else still_there_ids << bookmark['id'] end end articles = JSON.parse($instapaper.request("/api/1/bookmarks/list?limit=#{25 - num_not_there}&have=#{still_there_ids.join(',')}").body) articles.each do |article| next unless article.has_key?('title') filename = article['title'] filename = "Article #{article['bookmark_id']}" unless article['title'] file_path = "#{filename}.html" File.open(File.join(path, file_path), 'w') do |f| f << $instapaper.request("/api/1/bookmarks/get_text?bookmark_id=#{article['bookmark_id']}").body end $db.execute("insert or ignore into bookmarks (id, title, status) values (?,?,?)", article['bookmark_id'], filename, 2) end `cd #{path} && find . -name "*html" -exec #{KINDLEGEN} {} \\;` `cd #{path} && find . -name "*html" -exec rm {} \\;`
Gnus configuration
Here is the configuration for Gnus.
(load-library "smtpmail")
(load-library "nnimap")
(load-library "starttls")
(setq gnus-select-method '(nntp "news.gmane.org"))
(setq gnus-secondary-select-methods
'((nnml "")
(nnimap "imap.gmail.com"
(nnimap-address "imap.gmail.com")
(nnimap-server-port 993)
(nnimap-authinfo-file "~/.imap-authinfo")
(nnimap-expunge-on-close 'never)
(nnimap-stream ssl))))
(setq smtpmail-starttls-credentials '(("smtp.gmail.com" 587 nil nil))
smtpmail-smtp-server "smtp.gmail.com"
smtpmail-default-smtp-server "smtp.gmail.com"
send-mail-function 'smtpmail-send-it
message-send-mail-function 'smtpmail-send-it
smtpmail-smtp-service 587
smtpmail-auth-credentials
'(("smtp.gmail.com" 587 "ykabhinav@gmail.com" nil)))
(add-hook 'gnus-topic-mode-hook 'gnus-topic-mode)
;(setq gnus-summary-rescan-group 'all)
;(setq gnus-permanently-visible-groups ".*INBOX")
(setq gnus-thread-sort-functions
'(gnus-thread-sort-by-most-recent-date
(not gnus-thread-sort-by-total-score)))
(gnus-demon-add-handler 'gnus-group-get-new-news 5 nil)
(cond (window-system
(setq custom-background-mode 'light)
(defface my-group-face-1
'((t (:foreground "Red" :bold t))) "First group face")
(defface my-group-face-2
'((t (:foreground "DarkSeaGreen4" :bold t)))
"Second group face")
(defface my-group-face-3
'((t (:foreground "Green4" :bold t))) "Third group face")
(defface my-group-face-4
'((t (:foreground "SteelBlue" :bold t))) "Fourth group face")
(defface my-group-face-5
'((t (:foreground "Blue" :bold t))) "Fifth group face")))
(setq gnus-group-highlight
'(((> unread 200) . my-group-face-1)
((and (< level 3) (zerop unread)) . my-group-face-2)
((< level 3) . my-group-face-3)
((zerop unread) . my-group-face-4)
(t . my-group-face-5)))
(setq-default
gnus-summary-line-format "%U%R%z %(%&user-date; %-15,15f %* %B%s%)\n"
gnus-user-date-format-alist '((t . "%Y-%m-%d %H:%M"))
gnus-summary-thread-gathering-function 'gnus-gather-threads-by-references
gnus-thread-sort-functions '(gnus-thread-sort-by-date)
gnus-sum-thread-tree-false-root ""
gnus-sum-thread-tree-indent " "
gnus-sum-thread-tree-leaf-with-other "|-> "
gnus-sum-thread-tree-root ""
gnus-sum-thread-tree-single-leaf "\-> "
gnus-sum-thread-tree-vertical "│")
(setq gnus-use-cache t)Configuring Gnus
Gnus for reading Gmane, but I think I need to do a bit more research in
terms of figuring out how to show mail, etc. Stay tune.
Thoughts on Using Emacs
I converted to Emacs all of seven months ago and I wish I started earlier. The editor is way powerful, it is a Lisp Machine for the rest of us. Considering its origins it makes sense. It’s an editor you build up to the way you think instead of stooping down to think the way the editor works.
I won’t recommend jumping full into the Emacs way using ERC, GNUS, VM, and all, but start off small. I jumped in by using org-mode.
Alos, learn something new everyday, learn Elisp and add small custom functionality. You will be spending most of your time in your editor if you are a coder so might as well learn its language. Always be sharpening your tools!
mutt with Emacs and BBDB
I wanted to really use Emacs and BBDB with my mail client and that’s partially why I spent so much time trying out different Emacs mail clients. But I can do the same with mutt and Emacs + BBDB and don’t have to bang my head at the slowness.
First step is to add the following to your .muttrc:
set edit_headers set editor = "emacsclient -a emacs"
This allows you to edit the headers when you are editing the message and also use emacsclient. This assumes that you are running Emacs in daemon mode and will start a new instance if it isn’t. I also set the mail-mode for the mutt compose buffers. So add the following in your .emacs file.
(start-server)
(add-to-list 'auto-mode-alist
'("\\.*mutt-*\\|.article\\|\\.followup"
. mail-mode))Now install BBDB I am using BBDB 3.0 which is still in development but quite stable. I have the following in my .emacs file.
(require 'bbdb)
(require 'bbdb-message)
(bbdb-insinuate-message)
(if bbdb-complete-mail
(define-key mail-mode-map "\C-t" 'bbdb-complete-mail))Now you will be able to use bbdb for completing names and have Emacs’s mail mode at your disposal when editing emails. Sort of like this post.
Emacs VM config for Gmail IMAP
Make sure that you have stunnel 4.33 or lower for the trunk version of VM. https://launchpad.net/vm
(add-to-list 'load-path "~/Dropbox/vm/lisp")
(require 'vm-autoloads)
(setq mail-user-agent 'vm-user-agent)
(setq vm-folder-directory "~/Mail")
(setq vm-primary-inbox "~/Mail/vminbox")
(setq vm-crash-box "~/Mail/inbox.crash.mbox")
(setq vm-imap-expunge-after-retrieving nil)
(setq vm-mutable-frames nil)
(setq vm-imap-account-alist
'(("imap-ssl:imap.gmail.com:993:login:ykabhinav@gmail.com:*" "gmail")))
(setq vm-primary-inbox "imap-ssl:imap.gmail.com:993:inbox:login:ykabhinav@gmail.com:*")
(setq vm-debug t)
(setq vm-keep-imap-buffers t)
(setq vm-imap-log-sessions t)
(setq vm-use-menus 1)
(setq vm-stunnel-program "stunnel")
(setq send-mail-function 'smtpmail-send-it
message-send-mail-function 'smtpmail-send-it
smtpmail-starttls-credentials
'(("smtp.gmail.com" 587 nil nil))
smtpmail-auth-credentials
(expand-file-name "~/.authinfo")
smtpmail-default-smtp-server "smtp.gmail.com"
smtpmail-smtp-server "smtp.gmail.com"
smtpmail-smtp-service 587
smtpmail-debug-info t)
(require 'smtpmail)
(setq mail-default-headers "From: Abhi Yerra <ykabhinav@gmail.com>\n")
(setq vm-mime-type-converter-alist
'(("text/html" "text/plain" "elinks -dump /dev/stdin")
("message/delivery-status" "text/plain")
("application/zip" "text/plain" "listzip")
("application/x-zip-compressed" "text/plain" "zipinfo /dev/stdin")
("application/x-www-form-urlencoded" "text/plain")
("message/disposition-notification" "text/plain")
("application/mac-binhex40" "application/octet-stream" "hexbin -s")))
(define-key vm-mode-map "#" 'vm-imap-synchronize)Return to mutt
I’ve sort of given up on Emacs mail clients. I really tried, I did. But all of them had their own problems and at the end of the day I need to get work done. I can’t worry whether my mail client is working. But I wanted a terminal interface to my mail client because I don’t really like using the gmail interface nowadays, it’s a bit slow and takes a while to load, etc.
I’ve decided to head back to mutt. One of the initial things I disliked about mutt was wrongfully thinking I couldn’t see my imap folders with the unread count. This was a wrong assumption. If you press ‘c’ in the pager or index views and then press you can see the folders/unread count. You can also press ‘y’ to get the same view.
set folder_format="%2C %t %N %8s %d %f"
Now the only problem I have is what to do about contact completion. That will be the next battle I face.
However, I do think mutt should incorporate a few more features within its fold. I mean I could totally see it having a Google Voice and Twitter integration.
Back to mutt
was expecting too much of email from Emacs. Though I do still want to
maintain my contact list within it via BBDB so I guess I'll be
focusing more time on that then VM.
Emacs MUA Shopping ((tags: emacs, vm)
I’ve been trying to find a perfect Emacs mail client and I am a bit disappointed. I thought Mew would be a good choice and to a large extent it was but it seems to be lacking any recent updates.
I’ve tried VM and initially I had high hopes but I am a bit stuck on of how to move messages between folders. Also, IMAP sort of sucks and it is even stated as such in their TODO that it sucks. But everything else seems to be quite nice about it. I like the integration with BBDB. I like how it displays messages and the virtual folders stuff seems pretty nifty.
I feel that VM probably has the most developer momentum right now and hopefully things move in a positive direction.
BBDB Sync Google Contacts
Since I am starting to use Mew I though I should start looking into the BBDB Old/ New.
But I store my contacts in Google because I love their Sync service. But all is not lost. I discovered googlecl which is a nifty commandline way to access many Google services.
This allows me to update my records with something like the following.
(mapcar
'(lambda (x)
(bbdb-create-internal (first x) nil nil nil (second x)))
(mapcar
'(lambda (x) (split-string x ","))
(process-lines "/usr/local/bin/google" "contacts" "list" "name,email")))NOTE: I will update with a more thorough code in the future.

