Setting up Gmail in Doom Emacs using mbsync and mu4e(erichgrunewald.com)
erichgrunewald.com
Setting up Gmail in Doom Emacs using mbsync and mu4e
https://www.erichgrunewald.com/posts/setting-up-gmail-in-doom-emacs-using-mbsync-and-mu4e/
49 comments
I am proficient with emacs, but I've never gotten to the point where org-mode clicks. Any advice?
Taking notes is the simplest. I started with outline-mode back in the day. Once you start taking notes you'll think: if only I had a way of cataloguing, tagging and prioritising tasks. Before long you'll start estimating tasks and creating agendas; then you'll want to tie in other parts of your life to your tasks and notes -- as nothing ever lives in a vacuum -- and... well, there you go. That's how it goes.
Create a .org file and start writing is really the simplest. If you never make it past that, you still have an effective note taking system in Emacs.
Create a .org file and start writing is really the simplest. If you never make it past that, you still have an effective note taking system in Emacs.
For me it was org-babel[1] that finally sucked me in. For example, I keep notes of various shell invocations as source blocks. Also various one-off SQL queries. Org-mode tables are surprisingly powerful too. Basically, a lot of things that I used to do occasionally are now contextualized and embedded in my notes.
[1] https://orgmode.org/worg/org-contrib/babel/intro.html
[1] https://orgmode.org/worg/org-contrib/babel/intro.html
And If you name tables in org-mode, you can reference them from shell with some simple functions, e.g. `fromOrg tsv Network.org:[MAC-List]`
That’s news to me. Would you please elaborate?
Tables are fairly easy to extract from org-files, even with some sed/awk function, e.g. `getOrgTable() { awk -vT=$1 '/^#\+NAME: /&&$2~T{f=1;next};/----/{next};!/^\|/{f=0};f' $2 ; }`
The above should work for most tables, and you get back a pipe-separated-values output, that you could further refine with e.g. http://harelba.github.io/q/ to get tsv,csv output support and SQL query capabilities.
I regularly SQL-Query with INNER JOINs over tables in various org-files stored on multiple servers in one commandline using this combination.
John Kitchin has a description for a "cleaner" way to do it here: https://kitchingroup.cheme.cmu.edu/org/2014/08/11/Using-org-...
My scripts are a dirty combination of ideas/concepts of both of the above, that work for me depending on situation/use-case, but that I should clean up one day, and gist someplace. I can give more info, if the above doesn't give enough pointers yet for you to dig yourself further towards "your solution".
The above should work for most tables, and you get back a pipe-separated-values output, that you could further refine with e.g. http://harelba.github.io/q/ to get tsv,csv output support and SQL query capabilities.
I regularly SQL-Query with INNER JOINs over tables in various org-files stored on multiple servers in one commandline using this combination.
John Kitchin has a description for a "cleaner" way to do it here: https://kitchingroup.cheme.cmu.edu/org/2014/08/11/Using-org-...
My scripts are a dirty combination of ideas/concepts of both of the above, that work for me depending on situation/use-case, but that I should clean up one day, and gist someplace. I can give more info, if the above doesn't give enough pointers yet for you to dig yourself further towards "your solution".
That’s plenty of information. Thank you!
I've been using Neomutt for some time now and I'm already contemplating ditching it. Fewer emails these days are purely text, viewing them brings me right back to the browser anyways. Never got into the Org-Mode hype for Doom Emacs. I do love Magit though, it remains the best git tool I've used to date, but the pain of wrestling with some of my lsp plugins (Looking at ESS) has made waste more time than anything.
mu4e is mail on steroids. If you need to automate anything you can grok a small script in an afternoon (much less if you are familiar with elisp) and have it work for you in ways no other mail agent will.
My need: write personalized exams for my (50-ish) students, send an email for each with his respective exam, and upon receiving the replies, save each attachment in its own directory.
Some perl+latex+mu4e+elisp ans voilà.
Edit: I refuse to use my Uni’s version of moodle for long-reply exams. Also using pdftools has changed my life.
My need: write personalized exams for my (50-ish) students, send an email for each with his respective exam, and upon receiving the replies, save each attachment in its own directory.
Some perl+latex+mu4e+elisp ans voilà.
Edit: I refuse to use my Uni’s version of moodle for long-reply exams. Also using pdftools has changed my life.
> mu4e is mail on steroids. If you need to automate anything you can grok a small script in an afternoon (much less if you are familiar with elisp)
It’s not a very good sales pitch to claim that automating anything needs a script and then even an afternoon to do it. Like I really hope that simple filter rules take at most a minute not hours. :-)
> and have it work for you in ways no other mail agent will.
How does it compare to Gnus?
It’s not a very good sales pitch to claim that automating anything needs a script and then even an afternoon to do it. Like I really hope that simple filter rules take at most a minute not hours. :-)
> and have it work for you in ways no other mail agent will.
How does it compare to Gnus?
I have found that running the html email through w3m generates usable text in almost all cases:
text/html; w3m -I %{charset} -T text/html; copiousoutput;
I'm very happy with mbsync and mu4e. The only drawback is that email fetching+indexing can be very slow (10 to 30 seconds), compared with the instantaneous gmail web interface. I'm not sure if this is poor configuration on my side, or some other bottlenecks (mbsync, mu indexing, IMAP protocol).
I've found the sequential syncing with mbsync to be a major bottleneck. These days instead of "mbsync -a" I have a script like this, using GNU Parallel:
parallel --bar -j0 mbsync "$@" <<EOF
mail1
mail2
mail3
EOF10-30 seconds sounds very slow indeed. It used to be slow for me (over 10 seconds against my university server), but I discovered it was because I had not deleted emails and propogated that to the server, I had only marked them for deletion (d), not deleted (D), and so there was some kind of syncing back and forth each time of quite many emails with mbsync. For me now it is always under 5 seconds for syncing and indexing, usually quicker.
I'm not sure if the gmail interface is instantaneous or if it's just using tricks to make it seem that way.
I've regularly waited up to a minute for an email I've just sent to arrive on someone else's account. A large part of this delay seems to be for some weird undo feature they've added which you can no longer fully disable.
I've regularly waited up to a minute for an email I've just sent to arrive on someone else's account. A large part of this delay seems to be for some weird undo feature they've added which you can no longer fully disable.
I discovered Doom Emacs a couple of months ago and it made me use emacs more often again. Doom Emacs is well made and makes configuration of emacs a lot faster and easier. I've noticed that a lot of file types have support from an LSP server and setting that up is also easy.
Long time ago, I used to read email using wanderlust on emacs. I remember I really liked it, it was very powerfull in handling email and it had really good imap (and offline) support. The only drawback was related to handling html email. Back then that was not that much of a problem because I used it for a lot of mailinglists which were typically text only, and the remaining email was often sent in both a text and a html version.
Anyone using email in emacs and has a good solution to deal with html email? Both receiving and sending? I kinda remember reading about people writing email in markdown and have it converted to html when sending it out.
Long time ago, I used to read email using wanderlust on emacs. I remember I really liked it, it was very powerfull in handling email and it had really good imap (and offline) support. The only drawback was related to handling html email. Back then that was not that much of a problem because I used it for a lot of mailinglists which were typically text only, and the remaining email was often sent in both a text and a html version.
Anyone using email in emacs and has a good solution to deal with html email? Both receiving and sending? I kinda remember reading about people writing email in markdown and have it converted to html when sending it out.
Mu4e can render HTML e-mails in Emacs (using Emacs’ shr render engine). Works well enough for most cases. When it doesn’t, there is a command for opening the message in your browser of choice. Re sending HTML: I personally think it’s a bad idea and don’t do it. But I think it‘s possible to write e-mails in org mode markup and them have them automatically converted to HTML. Probably easy to build something similar for Markdown, if it doesn’t exist yet.
You can indeed write an email in org-mode and convert it to html before sending it out: https://github.com/cashweaver/dotfiles/blob/main/config/doom....
I think this is the write-up I pulled these functions from: https://kitchingroup.cheme.cmu.edu/blog/2016/10/29/Sending-h...
I think this is the write-up I pulled these functions from: https://kitchingroup.cheme.cmu.edu/blog/2016/10/29/Sending-h...
Has anyone switched from Spacemacs to Doom Emacs? Are there special advantages Doom offers that make this switch worthwhile?
I can confirm a lot of what has already been said. I've used spacemacs since 2017, have contributed to it, and have written several private layers for it, such as a custom org layer.
But I switch to Doom emacs literally two weeks ago and I'm never going back. I find doom to be much faster not just startup time but in general responsiveness (one major exception: hitting SPC and waiting for key binding tips to show seems to be much slower in doom than spacemacs). Other than that I found doom to be much easier to configure and much more configurable.
The major selling point for me was that in doom emacs there is much much less ceremony in wrapping a new package in a module, and more importantly it is _easier_ to reuse documentation for those packages because most packages provide configuration examples using `use-package` anyway. My only hangup was less documentation around writing custom modules which took me about a day to figure out (I would advise to just read some of the source code or mimic a module that doom ships with).
YMMV, but if you feel like you've attained a certain level of emacs maturity, I would 100% recommend trying it out. Be prepared that some modules, like the haskell module, don't ship with a lot of keybindings so you'll have to set those up yourself.
But I switch to Doom emacs literally two weeks ago and I'm never going back. I find doom to be much faster not just startup time but in general responsiveness (one major exception: hitting SPC and waiting for key binding tips to show seems to be much slower in doom than spacemacs). Other than that I found doom to be much easier to configure and much more configurable.
The major selling point for me was that in doom emacs there is much much less ceremony in wrapping a new package in a module, and more importantly it is _easier_ to reuse documentation for those packages because most packages provide configuration examples using `use-package` anyway. My only hangup was less documentation around writing custom modules which took me about a day to figure out (I would advise to just read some of the source code or mimic a module that doom ships with).
YMMV, but if you feel like you've attained a certain level of emacs maturity, I would 100% recommend trying it out. Be prepared that some modules, like the haskell module, don't ship with a lot of keybindings so you'll have to set those up yourself.
You can fix the delay for which key using by modifying the following variable:
(setq which-key-idle-delay 0.1)
(setq which-key-idle-delay 0.1)
Just did this and it sure works.
Thanks for your answer.
I switched. Doom starts much faster and feels generally snappier and more minimal. I like the configuration system. I use this Spacemacs module for Doom[0] to retain (most of) the Spacemacs keybindings and some of the functionality.
[0] https://github.com/chenyanming/spacemacs_module_for_doom
[0] https://github.com/chenyanming/spacemacs_module_for_doom
I did, I used spacemacs for a few years, but switched to Doom at the beginning of 2020. It is dramatically faster in start up and usage. I also find the configuration a lot easier and straightforward. I did the initial switch because Java LSP was stalling out spacemacs constantly.
I tried, when Spacemacs was somehow broken on my system. I didn't find anything better on Doom, and fixing Spacemacs was easier than switching to Doom.
Using Doom, most of your muscle memory from Spacemacs is gone because not everything is configured the same (that is of course normal). But not all, and that make the experience weird to me.
I also feel like Spacemacs is growing faster and has more feature than Doom, because more dev's are working on Spacemacs. But I might be wrong about that.
I also feel like Spacemacs is growing faster and has more feature than Doom, because more dev's are working on Spacemacs. But I might be wrong about that.
I got into to emacs again about six months ago. Having not used it for about a decade, Spacemacs was a wonderful introduction to what is possible in Emacs today. I’ve since moved on to a personal straight/use-package based init with a relatively minimal package set centered on the vertico/embark ecosystem, but I remain fond of spacemacs and still look to it for ideas.
Really, I’m surprised we haven’t seen Prot on the front page here.
Really, I’m surprised we haven’t seen Prot on the front page here.
I’ve gone in this direction after using Spacemacs for a couple of years. I’ve found
- Startup speed much quicker though I do use emacsclient so this is only an issue on first launch
- Overall feels a lot more snappy
- Simpler and more straight forward configuration
- Updates seem to break less (though sadly this isn’t reduced to zero, just reduced).
- Startup speed much quicker though I do use emacsclient so this is only an issue on first launch
- Overall feels a lot more snappy
- Simpler and more straight forward configuration
- Updates seem to break less (though sadly this isn’t reduced to zero, just reduced).
Has anyone here used both offlineimap and mbsync and can speak to how these two compare?
I switched to mbsync from offlineimap about four years ago. I was having issues with offlineimap hanging during sync, as well as slow performance in general.
I switched to mbsync and have had zero issues since. Just set it and forget it.
I switched to mbsync and have had zero issues since. Just set it and forget it.
mbsync tends to be more efficient and robust. With Gmail it has no problems. With outlook IMAP it has problems with some emails occasionally. I haven’t figured out how to sync Maildir with Office365 other than cloud-mdir-sync which is inefficient for large mail boxes.
I switched from offlineimap to mbsync. I can't remember the details, but I remember having lots of various issues with offlineimap but I can't remember any issues with mbsync other than once I think I had to change something in my config after an upgrade.
You might want to store application specific passwords in the Keychain.
Here is how to do it in .mbsyncrc
https://gist.github.com/eddieh/8c853c6cf8ffb3ad87e0720eb50f8...
And here is how to do it in .msmtprc
https://gist.github.com/eddieh/5b4df4a8a98ea202e6ebb020871b0...
Here is how to do it in .mbsyncrc
https://gist.github.com/eddieh/8c853c6cf8ffb3ad87e0720eb50f8...
And here is how to do it in .msmtprc
https://gist.github.com/eddieh/5b4df4a8a98ea202e6ebb020871b0...
And for Gmail specifically you might want to look at the mbsync option PipelineDepth
As noted in the .mbsyncrc gist:
> The value is "used to limit average bandwidth consumption (GMail may require this if you have a very fast connection)." Turns out the same is true for iCloud. There's probably some optimized value greater than "1" that can be used, but this works for me.
I actually have it set to 5 now.
As noted in the .mbsyncrc gist:
> The value is "used to limit average bandwidth consumption (GMail may require this if you have a very fast connection)." Turns out the same is true for iCloud. There's probably some optimized value greater than "1" that can be used, but this works for me.
I actually have it set to 5 now.
Would be great if there was an option for those of us that have enabled the "2-step verification" setting for our account. :(
Does https://support.google.com/accounts/answer/185833?hl=en not work?
It is silly, as I think that is disabled if you have "Advanced Protection" turned on. Which... is what I thought they now called 2-step. Seems it is 2-step+?
So, yeah, "Advanced Protection" offerings would be good. Looks like I could make my own app that uses open id to connect?
So, yeah, "Advanced Protection" offerings would be good. Looks like I could make my own app that uses open id to connect?
This is possible with mbsync. Take a look at Archlinux wiki page for mbsync.
The page I found assumes you can get an app password. Those are disabled in advanced. Am I looking at wrong page?
I used mbsync like this:
https://wiki.archlinux.org/title/Isync#Using_XOAUTH2
On Fedora, I think I had to dnf install "isync" (for mbsync) and "libkgapi" (for the SASL XOAUTH2 plugin) and also pip install "oauth2token".
You'll need to get credentials from Google. I think this describes how: https://developers.google.com/identity/protocols/oauth2
If you have a plain gmail account (instead of Google Workspace, or whatever it's called now), one irritating thing is that you have to renew the access tokens (using oauth2create) once a week.
On Fedora, I think I had to dnf install "isync" (for mbsync) and "libkgapi" (for the SASL XOAUTH2 plugin) and also pip install "oauth2token".
You'll need to get credentials from Google. I think this describes how: https://developers.google.com/identity/protocols/oauth2
If you have a plain gmail account (instead of Google Workspace, or whatever it's called now), one irritating thing is that you have to renew the access tokens (using oauth2create) once a week.
I'm comfortable with Emacs for programming, Org and RSS.
I've tried using email from Emacs a number of times, but never had a setup that I stick with.
All modes that used IMAP would block.
notmuch.el was nice but distributed read-state between two DBs, which felt brittle.
I use Thunderbird. I'm keen to use email from Emacs, if I can find a decent solution.
I've tried using email from Emacs a number of times, but never had a setup that I stick with.
All modes that used IMAP would block.
notmuch.el was nice but distributed read-state between two DBs, which felt brittle.
I use Thunderbird. I'm keen to use email from Emacs, if I can find a decent solution.
Do you mean for fetching? I've got offlineimap set up for mine, and it's primarily a background task, because Emacs is not dealing with it directly.
Using mu4e, it is more of an interface to that local store with its built-in search API.
Using mu4e, it is more of an interface to that local store with its built-in search API.
Better to just have two or more Emacs processes running. One for programming, one for email, one for Elfeed. If I had to check email in Emacs, I might do it this way, but like you I just use Tbird and Emacs.
If anyone wants to try email in Emacs, two very productive options I used, years ago were:
* VM https://www.nongnu.org/viewmail/
* Gnus https://www.gnus.org/
I don't know about the current security of these.
I used both of them with BBDB contacts database (which I found much more useful than anything that JWZ's employer did with addressbooks, then or since): https://www.jwz.org/bbdb/ https://www.emacswiki.org/emacs/CategoryBbdb
* VM https://www.nongnu.org/viewmail/
* Gnus https://www.gnus.org/
I don't know about the current security of these.
I used both of them with BBDB contacts database (which I found much more useful than anything that JWZ's employer did with addressbooks, then or since): https://www.jwz.org/bbdb/ https://www.emacswiki.org/emacs/CategoryBbdb
Great write up! I would suggest using OAuth2 rather than the Less Secure App settings (https://cashweaver.com/blog/read-email-in-emacs-with-mbsync-...).
I also wrote up a guide for using notmuch, which has quickly become my preferred emacs email solution: https://cashweaver.com/blog/read-email-from-gmail-with-notmu...
I also wrote up a guide for using notmuch, which has quickly become my preferred emacs email solution: https://cashweaver.com/blog/read-email-from-gmail-with-notmu...
sweet, thanks. looks like you pretty much got there before me. i wonder what it is about doing this that makes people want to blog about it ...
notmuch is great, too
If you are not on GMail but have SSH access to your Maildir, a nice (faster) alternative to mbsync or offlineimap is "Sync Mail Dir", short smd: http://syncmaildir.sourceforge.net/
It works well, but unfortunately it needs a new maintainer: https://github.com/gares/syncmaildir
Org-mode. Capturing whatever into the appropriate note is wonderful. These days about 80% of the time my answer to why Emacs is org-mode. Not that there aren't other great note taking options, but I've yet to find one that integrates so fully with, well, everything.
For all the doubters, I was right there with you 6 months ago. Who cares about yet another markdown clone am I right? Perhaps someone else can explain better, but in my case no amount of explanation really helped, I didn't get it until I did it.