Launch HN: Compose.ai (YC W21) – AI-powered extension to speed up email writing
63 comments
I like the idea, but have the same concerns which prevent me from using Grammarly. I understand the challenge of doing this locally. But I just can’t get myself to install something that siphons off essentially every keystroke I make.
On a more positive note: is there something that detects or warns against hostile tone to help write in a friendly and non-hostile way? :)
On a more positive note: is there something that detects or warns against hostile tone to help write in a friendly and non-hostile way? :)
Would you use a chrome extension that stored the auto-complete model locally, and periodically sent anonymous statistical updates? Your keystrokes never leave your machine (and the model would work offline).
Federated machine learning is an area of my research.
Federated machine learning is an area of my research.
That's really interesting. Let me know if you'd want to talk to me or Michael sometime about it. With the models we run currently, you'd really have to have a GPU to run locally and get a lot of utility. I'm curious if you have some thoughts on how to run these large language models on edge devices.
I'm wilson@ our website (trying to avoid too much spam from bots).
I'm wilson@ our website (trying to avoid too much spam from bots).
I sent y'all an email, but figured I'd re-post here for any curious hackers. I spent two years obsessed with autocomplete for mobile/edge use cases.
The first step is to get any functional offline model (1), then prune/project a large language model's representation until you can perform on-device inference (2). You can calculate variance, hit / miss statistics for a body of text and model proposals (3), which you can feed into a ranking model (4) for an extra layer of personalization or use to re-balance the Euclidean projection of your model's layers (4) to optimize for sparseness.
1) Locally store a Trie data structure, where keys are n-grams of user input
Surprisingly effective, considering most business communication uses a limited vocabulary. If your users are submitting less than 10,000 unique English words (skip words removed) per day, try this out.
One thing I really liked about the Trie approach is that corporate jargon appears in real-time, since the "model" is just a data retrieval algorithm. You don't need to modify a vocabulary and re-train/fine-tune a neural network to achieve personalization.
The downside is that you're limited to bi/tri-grams before performance degrades, although YMMV. Auto-completing bi/tri grams does feel tedious after a while.
2) Fine-tune and prune a large language model, then make it sparse
I noticed y'all offer some degree of personalization. Have you tried pruning or compressing your model after fine-tuning? The exact technique will depend on your base model's architecture but in general, try using a sparser representation.
Use accelerators designed to operate on sparse representations, for example TensorFlow XNNPack's sparse operations. XNNPack is a backend engine that opens up native hardware acceleration options in WebAssembly, so you can accelerate inference using the client's GPU (if available).
3) Collect permutation variance and hit/miss statistics
The exact technique/algorithm will depend on your model architecture, but for example matched averaging is a way to express the average number of neighborhood permutations with respect to the input dataset. In other words, the client sends statistics about predictions in Euclidean space, not your literal keystrokes.
4) Use matched averaging to adjust model cardinality or train an additional ranking model
The statistics collected by step 3 can be used to train a personalized ranking model, with the goal of re-ranking the proposals from step 2.
You can also use these statistics to introspect the "embedding space" of a language model, with the goal of identifying compression/pruning opportunities to improve the model's real-time performance. Reducing cardinality in the embedding projection has an outsize impact on inference speed, and you can usually drop most of the language model after observing the range of language used by the client.
You can also used matched averaging to compare hidden <-> hidden weights between many with Euclidean distance measurements (like cosine distance).
This is WAY more than I originally intended to write - but I hope this helps!
The first step is to get any functional offline model (1), then prune/project a large language model's representation until you can perform on-device inference (2). You can calculate variance, hit / miss statistics for a body of text and model proposals (3), which you can feed into a ranking model (4) for an extra layer of personalization or use to re-balance the Euclidean projection of your model's layers (4) to optimize for sparseness.
1) Locally store a Trie data structure, where keys are n-grams of user input
Surprisingly effective, considering most business communication uses a limited vocabulary. If your users are submitting less than 10,000 unique English words (skip words removed) per day, try this out.
One thing I really liked about the Trie approach is that corporate jargon appears in real-time, since the "model" is just a data retrieval algorithm. You don't need to modify a vocabulary and re-train/fine-tune a neural network to achieve personalization.
The downside is that you're limited to bi/tri-grams before performance degrades, although YMMV. Auto-completing bi/tri grams does feel tedious after a while.
2) Fine-tune and prune a large language model, then make it sparse
I noticed y'all offer some degree of personalization. Have you tried pruning or compressing your model after fine-tuning? The exact technique will depend on your base model's architecture but in general, try using a sparser representation.
Use accelerators designed to operate on sparse representations, for example TensorFlow XNNPack's sparse operations. XNNPack is a backend engine that opens up native hardware acceleration options in WebAssembly, so you can accelerate inference using the client's GPU (if available).
3) Collect permutation variance and hit/miss statistics
The exact technique/algorithm will depend on your model architecture, but for example matched averaging is a way to express the average number of neighborhood permutations with respect to the input dataset. In other words, the client sends statistics about predictions in Euclidean space, not your literal keystrokes.
4) Use matched averaging to adjust model cardinality or train an additional ranking model
The statistics collected by step 3 can be used to train a personalized ranking model, with the goal of re-ranking the proposals from step 2.
You can also use these statistics to introspect the "embedding space" of a language model, with the goal of identifying compression/pruning opportunities to improve the model's real-time performance. Reducing cardinality in the embedding projection has an outsize impact on inference speed, and you can usually drop most of the language model after observing the range of language used by the client.
You can also used matched averaging to compare hidden <-> hidden weights between many with Euclidean distance measurements (like cosine distance).
This is WAY more than I originally intended to write - but I hope this helps!
Thanks for the thoughts here. Will follow-up over email!
It sounds much more privacy respecting. I guess it depends on what’s included in those statistical updates and how much trust I can give the extension. For example for not getting hacked or bought by hackers (see the great suspender as an example).
Edit: I don’t even use Chrome :) but that’s not the main point, but perhaps shows where I am personally on the privacy scale?
Edit: I don’t even use Chrome :) but that’s not the main point, but perhaps shows where I am personally on the privacy scale?
For the sake of discussion, what about an open source decentralized blockchain that shows that your data is always encrypted and only readable by hosted apps that are open source that you approve to have access to your data?
I’m not sure if you’re joking or not. I tend to believe that simple designs improve security, so this sounds like the polar opposite.
But I guess I’m not the main target audience. Given the succees of Grammarly and other privacy invasive apps, people are happy to give away their data for convenience. The HN crowd might be a bit different however. Winning this crowd might or might not be an indication of success.
But I guess I’m not the main target audience. Given the succees of Grammarly and other privacy invasive apps, people are happy to give away their data for convenience. The HN crowd might be a bit different however. Winning this crowd might or might not be an indication of success.
https://languagetool.org/ is an OSS Grammarly competitor that you can self-host.
Took a while to find https://languagetool.org/dev almost everything else points to a commercial service.
Thanks for the comment. We do give users the ability to easily disable the extension temporarily or on websites that they might have particular concerns about.
We don’t currently have a way to detect hostile tones, but we do have a way to rephrase sentences to be more friendly — https://www.loom.com/share/703d223a40f34447897d1736c6dc8c91.
We don’t currently have a way to detect hostile tones, but we do have a way to rephrase sentences to be more friendly — https://www.loom.com/share/703d223a40f34447897d1736c6dc8c91.
Google actually managed to ship a grammar checker that works entirely locally on its devices. Here's a technical blog on how it's done: https://ai.googleblog.com/2021/10/grammar-correction-as-you-...
>On a more positive note: is there something that detects or warns against hostile tone to help write in a friendly and non-hostile way? :)
Similar idea for public figures: ML that can detect phrases that can be taken out of context easily.
Similar idea for public figures: ML that can detect phrases that can be taken out of context easily.
You wrote "Your data is end-to-end encrypted", but your webpage doesn't mention that anywhere (just says you won't sell users' data) and it kinda seems impossible as you need to be able to see what I write to give suggestions...
What I mean by end-to-end encrypted here is during transportation and at rest (not that it is never decrypted in memory). As you are pointing out, it obviously has to be in memory as clear text at some point. But that is being done on a private VPC and only very briefly.
That's not end-to-end encrypted then. In an end-to-end encrypted system the server never sees the plaintext. I'm not saying that this is a must-have for something you are making, but please do not mislead people into a false sense of privacy.
I guess he meant end to end where the other client is, well, the server. Haha
I believe that you didn't intend to mislead, but "end-to-end encryption" is a term of art and you don't meet the definition. It would only apply to your product if you had some sort of synchronization between clients and you didn't have the keys to decrypt that data.
What you're describing is "encrypted in transit" and "encrypted at rest". Encryption in transit is table stakes for any software today, not a differentiator. As for encryption at rest, if I saw that I'd wonder why my email content is ever at rest on your servers.
Again, I don't think you meant to mislead, but it's important to use terms of art accurately, especially when marketing to professionals in the field.
What you're describing is "encrypted in transit" and "encrypted at rest". Encryption in transit is table stakes for any software today, not a differentiator. As for encryption at rest, if I saw that I'd wonder why my email content is ever at rest on your servers.
Again, I don't think you meant to mislead, but it's important to use terms of art accurately, especially when marketing to professionals in the field.
Any plans to use homomorphic encryption eventually? low priority rn but would be v nice to have
We've looked at things like this or federated learning as well. Definitely will keep our eye on it as we continue.
[deleted]
My contrarian view is that instead of tools to help us churn out mundane crap, maybe we should work to eliminate the need to do mundate, repetative stuff.
Choosing words to write an email is not difficult, most of the time. It's a bit like AI code helpers. The syntax and the small details are not the challenge. The greater organization, design, and sense of direction are the hard problems.
Google already provides surprisingly accurate 2-3 word suggestions while writing documents and emails.
Wow. I just watched one of the linked videos. I would hate that. Who writes text without even a basic sense of the words they want to say? My limitation is typing speed, not thinking speed. Having to read popups every one or two words and decide if that's what I want to say is not only much slower but also really distracting. Not only does one need to try to keep up with a thought they want to express, but they also have to sift through and avoid bad suggestions. This is a nightmare scenario, like trying to write a paper on a phone with text suggestions.
Choosing words to write an email is not difficult, most of the time. It's a bit like AI code helpers. The syntax and the small details are not the challenge. The greater organization, design, and sense of direction are the hard problems.
Google already provides surprisingly accurate 2-3 word suggestions while writing documents and emails.
Wow. I just watched one of the linked videos. I would hate that. Who writes text without even a basic sense of the words they want to say? My limitation is typing speed, not thinking speed. Having to read popups every one or two words and decide if that's what I want to say is not only much slower but also really distracting. Not only does one need to try to keep up with a thought they want to express, but they also have to sift through and avoid bad suggestions. This is a nightmare scenario, like trying to write a paper on a phone with text suggestions.
Whenever I'm coding all those auto complete popups almost 100% obscure some visual information I'm keyed on to complete the task.
I absolutely don't understand who those popups are designed for.
I absolutely don't understand who those popups are designed for.
The extension works on hackernews but it doesn't work on any site that's marked experimental (not even on github, shown as example in the userguide), it always shows "Compose Now Not Available". also tried Compose.AI to compose this message but all suggestions just threw away some of the info
Hey, Wilson from Compose.ai here.
I just checked into it. I think the misunderstanding is that force enable only enables autocomplete. Compose Now and Rephrase still misbehave on the majority of websites and make them hard to use, so we didn't include them in the force enable functionality. There's a note about it on the Notion doc Michael linked, but that isn't clear enough in the product. Thanks for letting us know! I'll fix that!
Let me know if you're actually not seeing Github show autocomplete either. I'd love to debug that, but I checked again and it seems to be working for me right now.
I'm curious what you mean by "also tried Compose.AI to compose this message but all suggestions just threw away some of the info"
I just checked into it. I think the misunderstanding is that force enable only enables autocomplete. Compose Now and Rephrase still misbehave on the majority of websites and make them hard to use, so we didn't include them in the force enable functionality. There's a note about it on the Notion doc Michael linked, but that isn't clear enough in the product. Thanks for letting us know! I'll fix that!
Let me know if you're actually not seeing Github show autocomplete either. I'd love to debug that, but I checked again and it seems to be working for me right now.
I'm curious what you mean by "also tried Compose.AI to compose this message but all suggestions just threw away some of the info"
Compose Now is only available on certain websites, but we are working on enabling it everywhere else. You can find a list of websites we are officially integrated with here: https://composeai.notion.site/Supported-Tools-0f3ee54d5ef04d.... When a website is "experimental", you can force enable the extension (autocomplete) there but may encounter some issues. Thanks for pointing out the Github example—we'll look into that.
"We will never sell your data and will only use it to fulfill your completions or improve your results."
Just to try and understand the nuance — "your results" is collective you here, right? As in, will things that Alice writes be used to make suggestions to Bob, or will they be ringfenced in an Alice-specific model?
Are inputs being used to iteratively train models & then discarded or are they being held around in a corpus (presume the latter, since you mentioned encryption at rest)?
How do you handle weeding out inadvertently storing/sharing sensitive info when training and suggesting? E.g. passport number, social security numbers, bank details etc.
Just to try and understand the nuance — "your results" is collective you here, right? As in, will things that Alice writes be used to make suggestions to Bob, or will they be ringfenced in an Alice-specific model?
Are inputs being used to iteratively train models & then discarded or are they being held around in a corpus (presume the latter, since you mentioned encryption at rest)?
How do you handle weeding out inadvertently storing/sharing sensitive info when training and suggesting? E.g. passport number, social security numbers, bank details etc.
Is this based on what I type, or what other people have typed? I type the same stuff week after week, and would like to make that more efficient. But my most repetitive stuff is specific to me, so it would have to be personalized for it to be worthwhile.
From watching the video above, it looked like the user spend a fair amount of time selecting autocomplete options, and could have typed the whole thing out in roughly the same amount of time. Are there examples that show more bang-for-buck?
From watching the video above, it looked like the user spend a fair amount of time selecting autocomplete options, and could have typed the whole thing out in roughly the same amount of time. Are there examples that show more bang-for-buck?
What I would like to see is an AI for reading emails. Specifically for one use case: When you get roped into a long existing email chain and now you're trying to catch up by reading bottom-up through 50 replies, each one with a full to/from/date/subject in different styles that you have to somehow ignore, while noticing one line (or word!) responses or understanding context.
Actually, it doesn't need much AI - just an intelligently written text- filtering and formatting script would do it. I've actually written simple scripts in the past that will take a long email or a bunch of them, and export them as an better formatted HTML-based Word Doc that I can read top-to-bottom and refer back to easily. But it would be awfully nice to have something built in.
I'm honestly amazed at all the email jockeys out there that deal with this daily and have no problem with it. It's been a serious problem for literally decades.
Is there an open source project that is just a library for authenticating and working with all the different email services out there? It would be nice to have a MailKit with some sort of basic SQLite based storage. Then devs could just work on the UI.
Actually, it doesn't need much AI - just an intelligently written text- filtering and formatting script would do it. I've actually written simple scripts in the past that will take a long email or a bunch of them, and export them as an better formatted HTML-based Word Doc that I can read top-to-bottom and refer back to easily. But it would be awfully nice to have something built in.
I'm honestly amazed at all the email jockeys out there that deal with this daily and have no problem with it. It's been a serious problem for literally decades.
Is there an open source project that is just a library for authenticating and working with all the different email services out there? It would be nice to have a MailKit with some sort of basic SQLite based storage. Then devs could just work on the UI.
I think there are a few options/solutions to this already - such as the product below:
https://www.spikenow.com/features/conversational-email/
https://www.spikenow.com/features/conversational-email/
Not sure if I fully understand the idea you are suggesting here. I have certainly thought quite a bit about:
1. Summarizing email
2. Extracting intents from emails
But it sounds like you are suggesting something a bit different.
But it sounds like you are suggesting something a bit different.
Did you use compose.ai to write this post?
Yes — and now, to reply to your comment :)
So it inserts emoticons? Is that based on a users history or derived from the corpus?
We have an emoticons feature as a work in progress and we are continuously experimenting with personalization!
You will want a way to disable them, or just shelve them indefinitely. I don't want to use or see emoticons in my email
aka no
This is actually useful for me. But don't think it understands the concept when you press ALT+G. I wanted to leave a comment expanding "tried it loved it" but it thinks I'm talking about food for some reason :/
https://i.imgur.com/NAKhwMo.png
Also I think you should really checkout github co-pilot's autocomplete. When writing readme files I see it often completes the whole sentence in gray text which is much faster to expand and easier to complete. Your popup is a bit distracting compared to the inline gray text of copilot/gmail.
https://i.imgur.com/BcddljD.png
https://i.imgur.com/NAKhwMo.png
Also I think you should really checkout github co-pilot's autocomplete. When writing readme files I see it often completes the whole sentence in gray text which is much faster to expand and easier to complete. Your popup is a bit distracting compared to the inline gray text of copilot/gmail.
https://i.imgur.com/BcddljD.png
Thanks for mentioning these. We'll look into what is happening with the context on Hacker News.
I am familiar with co-pilot and I appreciate the suggestion and think it's possible you're right. We actually used to have gray text which you can enabled in certain sites still if you click the extension icon in the top left of your browser: https://imgur.com/a/90wDnvN
We found that it was very difficult to integrate that grey text with many websites in the way you might expect it to work. We are still exploring what a happy medium might look like.
I am familiar with co-pilot and I appreciate the suggestion and think it's possible you're right. We actually used to have gray text which you can enabled in certain sites still if you click the extension icon in the top left of your browser: https://imgur.com/a/90wDnvN
We found that it was very difficult to integrate that grey text with many websites in the way you might expect it to work. We are still exploring what a happy medium might look like.
Just want to say this looks amazing, and I’ll both install and subscribe as soon as I get back to my desk.
If this is even 50% as effective on my email writing as CoPilot has been on my code writing, then you’ve got a customer for life.
Congrats on the launch!
If this is even 50% as effective on my email writing as CoPilot has been on my code writing, then you’ve got a customer for life.
Congrats on the launch!
After watching the two videos you shared I don't believe a second that you are actually saving time.
It seems like 95% of the time the suggestions are irrelevant and actually harm the writing speed by forcing you to read bad suggestions.
I can see some value in the second video with the kind of template system, but for the rest no thank you.
It seems like 95% of the time the suggestions are irrelevant and actually harm the writing speed by forcing you to read bad suggestions.
I can see some value in the second video with the kind of template system, but for the rest no thank you.
I don't see it mentioned anywhere - can you add an Autocomplete Provider for android based on this so I can use it on my phone?
also really really not a fan of requiring a subscription just to look around the web app; Hard to see what it can do.
also really really not a fan of requiring a subscription just to look around the web app; Hard to see what it can do.
How does your extension compare to the AI-powered autocompletion in Gmail?
There is some similarity and there were some inspirations from Gmail to be sure. For example, there were many users of Superhuman or Front who really disliked the fact that they wouldn't be able to use autocompletions in their email client (we work on Superhuman in the browser and Front support is coming soon).
We do find that we are able to predict significantly more text than Gmail is able to.
We do find that we are able to predict significantly more text than Gmail is able to.
Hey Michael,
I love this. It's so cool! I just wish it worked in my slack app on Linux too. But I know you're working on that. Or at least that's what compose.ai thinks ;)
I love this. It's so cool! I just wish it worked in my slack app on Linux too. But I know you're working on that. Or at least that's what compose.ai thinks ;)
Thanks! You can upvote those integrations so we can prioritize them: https://composeai.canny.io/integrations. We can then keep you posted.
I've used Copy.Ai in the past, how are you guys different?
The biggest difference is our Chrome extension that does autocompletion across a large number of websites and things like full email completion (see the videos in the main post).
Are you planning to create a Firefox extension?
We're looking into adding support for other browsers but don't have a concrete timeline yet. You can upvote that here https://composeai.canny.io/integrations/p/add-support-for-fi... so we can keep you posted.
Do you offer an API?
We do have an API that we are offering to select companies and for specific use-cases. You can apply for access here: https://1hzsaknho9a.typeform.com/to/fGEQgXy7?typeform-source...
what took you guys so long to go to market?
We actually just didn't do a Launch HN post till now.
I was in Mexico typing my 10th reply to a mundane email when it hit me: people shouldn’t be typing every letter of every word — especially not the same things over and over again. That got me thinking about building a tool that helps people type faster anywhere online and ultimately save them time—something that was easier said than done. It was quite challenging to balance the speed needed for real-time predictions with the model complexity needed for high quality predictions. Also, it was surprisingly challenging to integrate with a ton of different websites – I learned that there are many, many different ways to implement a “text box,” and experienced first hand, the joys of integrating with things like shadow dom and iframes.
I have been thinking about how to communicate with computers since I was in maybe 6th grade when I would literally dream about it. I remember I had a notebook in which I tried to analyze the linguistic structure of English to teach my computer English (not a great approach in retrospect). Later on, I worked in ML and AI at IARPA, Amazon, Disney, and BCG before starting an ML consulting firm. The trends in NLP finally caught up to my dream (LLM, distillation, fine-tuning, transformers, GPUs), so working on Compose AI seemed like the next logical step.
We are working to create a more holistic, AI-first writing solution. For now, our chrome extension has autocompletion, rephrasing (shorten, lengthen, change mood), and full email/reply generation functionalities. As you’re typing, the text is streamed to our backend servers, where we have relatively large transformers models that are tuned for specific scenarios.
If you need help with long-form content or the creative process, our web app (https://web.compose.ai) can generate product descriptions, blog outlines, sales pitches and more. Note: our web app asks for credit card on sign-up but includes a free 7-day trial.
For our Chrome Extension, we make money using a freemium subscription model. Your data is end-to-end encrypted and encrypted at rest. We will never sell your data and will only use it to fulfill your completions or improve your results.
Here are a couple demo videos of the extension generating responses to a Hacker News email:
https://www.loom.com/share/c36ec67b0a3a466aaa2ec5ee2399cb15
https://www.loom.com/share/44cf0f5893094d1191143b9f6fe11a5d
Thanks for reading! Would love to have some conversation with folks in the comments section below. Please share any feedback/ideas/experience!
Here are our open jobs: https://at.compose.ai/jointheteam
You can download Compose AI from the Chrome web store: https://at.compose.ai/download