Google applying to patent deep neural network (LSTM) for machine translation(freepatentsonline.com)
freepatentsonline.com
Google applying to patent deep neural network (LSTM) for machine translation
http://www.freepatentsonline.com/y2016/0117316.html
76 comments
I don't know what Google's reason is for this patent, but defensive patents are really common these days, and I think just about any big company has lawyers saying "patent everything you can, since you need a big patent portfolio for defensive purposes". Point is, just because Google is filing this patent doesn't mean they intend to stop others from using this approach. A link to a patent application isn't enough context to know.
If Google intends to patent defensively, they should immediately pledge on this patent. One thing I find very unfortunate, is while Google claims they're only really intended to use their software patents defensively, they've pledged not to with only a tiny number of their patent library.
One quick change in business strategy could turn Google into the world's largest patent troll.
One quick change in business strategy could turn Google into the world's largest patent troll.
Maybe that's their backup plan if the internet run by ads continues its decline
Continues its decline?
I thought that decline what only in a certain metric or ad type
Maybe HTML in-browser ads, but native, in-flow software ads and demand-based "rewarded" ads basically prop up the entire mobile software (especially gaming) market. There's only growth there, no slowing.
> One quick change in business strategy could turn Google into the world's largest patent troll
Is there any big tech company for which this statement does not apply? Or do you hold Google to a higher standard?
Is there any big tech company for which this statement does not apply? Or do you hold Google to a higher standard?
I think google invites this standard with the mission statements of "organize the world's information and make it universally accessible and useful" and "Don't be evil"
Most people do hold Google to a higher standard. As such, many people I know would hesitate to even recognize the possibility of Google doing this.
I, on the other hand, consider Google holding a patent not significantly different than Oracle holding a patent.
I, on the other hand, consider Google holding a patent not significantly different than Oracle holding a patent.
Twitter.
> defensive patents are really common these day
I had a discussion about this with an IBM representative 15 years ago at a symposium at Heinrich-Boell-Stiftung in Berlin: IBM's point was that one can always join their patent-pool (of defensive patents) which means you give them a free license to use your patents and you are free to use their "defensive" patents. This is completely broken: You never know which patents become relevant and any new player already lost because one just cannot keep up with a company that can extort a free license from you and then dump a few (hundred) million into development based on that.
I had a discussion about this with an IBM representative 15 years ago at a symposium at Heinrich-Boell-Stiftung in Berlin: IBM's point was that one can always join their patent-pool (of defensive patents) which means you give them a free license to use your patents and you are free to use their "defensive" patents. This is completely broken: You never know which patents become relevant and any new player already lost because one just cannot keep up with a company that can extort a free license from you and then dump a few (hundred) million into development based on that.
I don't know if this is the reason, but I think Google started getting a lot more paranoid about hoarding patents when Apple and Microsoft started going after Android OEMs in the early years. Google didn't really have any "counter-offensive" patent strategy then, which is why it went on a patent buying spree back then, although most of the available ones also got bought by Apple and Microsoft through Rockstar and so on.
So best case scenario, Google doesn't want to be caught with its pants down regarding patents. Worst case, it wants to "own" deep learning, so that nobody can really compete with them. Although I think that would be a little in conflict with their strategy to open source tensorflow.
To really figure out on which side Google is now playing we'll have to see how they respond to future patent reforms, and whether they join Microsoft and IBM to once again kill those reforms, or support the reforms to abolish software patents or drastically reduce their damage.
So best case scenario, Google doesn't want to be caught with its pants down regarding patents. Worst case, it wants to "own" deep learning, so that nobody can really compete with them. Although I think that would be a little in conflict with their strategy to open source tensorflow.
To really figure out on which side Google is now playing we'll have to see how they respond to future patent reforms, and whether they join Microsoft and IBM to once again kill those reforms, or support the reforms to abolish software patents or drastically reduce their damage.
> Worst case, it wants to "own" deep learning, so that nobody can really compete with them. Although I think that would be a little in conflict with their strategy to open source tensorflow.
Only a little. Releasing all the models and frameworks helps advance the field, helps with finding people to recruit, helps with integrating them into teams, and so on. This is why so many giants find it in their own self-interest to contribute to FLOSS these days.
Competition-wise, as is often said, Google has all the data. If for every deep learning advance they make $1 and the competitors make $0.95, they win. Patents here are quite helpful: you may make a neat translation app using some new tricks, and then discover when you go to commercialize it that oops, Google's patented 'using neural nets for translation'. Then you either quit, get sued, get bought, or give them most of your profits.
Only a little. Releasing all the models and frameworks helps advance the field, helps with finding people to recruit, helps with integrating them into teams, and so on. This is why so many giants find it in their own self-interest to contribute to FLOSS these days.
Competition-wise, as is often said, Google has all the data. If for every deep learning advance they make $1 and the competitors make $0.95, they win. Patents here are quite helpful: you may make a neat translation app using some new tricks, and then discover when you go to commercialize it that oops, Google's patented 'using neural nets for translation'. Then you either quit, get sued, get bought, or give them most of your profits.
I'd rather google patent them than some other sue-happy company who will abuse them. Who knows, maybe they will open source the patent.
I felt similarly about Sun, but then Oracle bought them.
Hypothetical: in any alternative universe, would you have felt any better if Microsoft or IBM had sued a defenceless Sun into bankruptcy?
Google has the largest market cap so that isnt a concern here
Google has the largest market cap right now so that isnt a concern here yet
FTFY
FTFY
Sun's market cap was once one of the largest.
[deleted]
I rather no company patent them than any do-no-evil company. Besides why do you think they won't use/abuse it? The whole point of spending so much ridiculous amount of money on a patent is to get exclusive rights to be the only one to use it. Isn't competition better than stagnation by a single company?
so, how do you ensure no company patent them?
Write the same documents but instead of submitting them to the patent office, release them on the Internet.
Demonstrate prior art? There are a lot of ways to do that without releasing production code.
Given historical prescedent, is existence of prior art actually an effective way in practice to prevent the granting of a patent?
Nope, but that doesn't matter - as enforcement is the concern, and there prior art shines. If the motive is purely defensive (and I don't include the menacing of a portfolio in that category), then it is the ideal move to make - as bad actors waste more energy filing poorly researched and easily refuted patents.
Probably doesn't prevent the granting of a patent, however it makes it impossible to enforce the patent. Mike and I published this in 1998 http://www.rage.net/wireless/wireless-howto.html . A Cisco legal team found this in 2008 and contacted me because the owners of patent #7035281 were coming after them. Doing a simple write-up of what I thought was obvious at the time - stick a wireless card into a Linux PC and have it route packets - may have saved all of us from having the wifi router in everyone's home restricted by patents. So whatever ideas you implement, be sure to blog about them and make sure archive.org gets a copy.
The problem is that other people will just create one or more patents which are around using your breakthrough algorithm in different contexts. The famous amazon one-click shopping patent as an example. The internet is the breakthrough, but because there's no patent on that, its easy to surround with patents that should be too obvious to be patents, but have legal teeth even so.
If you patent the core idea, the other patents become a lot less useful. (Not that I think Google is thinking this way. It's just a PR problem to them. When no one is looking, I bet they do whatever they can to get as much money/power as they can.)
If you patent the core idea, the other patents become a lot less useful. (Not that I think Google is thinking this way. It's just a PR problem to them. When no one is looking, I bet they do whatever they can to get as much money/power as they can.)
It is a terrible situation, and putting out prior art certainly doesn't fix that - but the alternatives courses of action are worse (assuming purely defensive interests). I think you're right though, if the internet was somehow patented then we wouldn't have one-click shopping patents... or much of anything really - we'd likely be having this discussion over a Minitel service.
[deleted]
Consider the development of Microsoft's attitude regarding software patents. There's no reason whatsoever to think these patents are in good hands with Google, especially considering that their Android licensing policies seem to be carbon copies of old Microsoft tactics.
http://arstechnica.com/business/2007/03/analysis-microsofts-...
http://arstechnica.com/business/2007/03/analysis-microsofts-...
Their goal as it seems up to this point is to get a monopoly on the framework and technology with opensourcing of their framework. Good for hires and keeping the competition in check. A patent will tighten their grip on ai.
>why y'all need to patent this then?
so someone else doesn't and trolls with it?
so someone else doesn't and trolls with it?
A defensive publication suffices to make it unpatentable.
Unfortunately, someone else can come along and patent a trivial, supposedly non-obvious "next step" invention, and the original inventor would have to pay to license the slightly improved version to remain competitive. With a patent on the main technology, they could probably negotiate a reciprocal deal at the very least.
What are rules for such a publication? I guess that something published in IEEE shouldn't be patentable and have my doubts about publishing on a blog, self-hosted, in Romanian. Where is the line according to the US law?
Any prior disclosure invalidates any patents.
To be sure, you submit the provisional patent application, and never file the utility. It's only $260 to file. After 1 year the provisional dies and you have a permanent record of the prior art.
In this case there is a provisional patent from 2014, and this application follows from that provisional.
In this case there is a provisional patent from 2014, and this application follows from that provisional.
> After 1 year the provisional dies and you have a permanent record of the prior art.
It's a permanent record, but it's not published. This is not a great plan.
It's a permanent record, but it's not published. This is not a great plan.
I've been reading more about this. Public disclosure vs. provisional.
In the US you get a year after public disclosure to file even a provisional. But not so if you want non-US patents. So you are closing some doors but keeping others open with public disclosure. Also, you are starting a 1 year clock.
A provisional is private; it does not count as public disclosure. A provisional is nothing more than a priority date, assuming what you have disclosed in the provisional itself is sufficient, and novel. You can even refile the same provisional every year as long as you believe the subject mater is still novel, but you get a new filing/priority date each time.
As for preventing a 3rd party from patenting the subject matter, either one is sufficient. However, if you publically disclose, only then you also get protection from a 3rd party who builds on your work. So in that case public disclosure is better than the 'secret' provisional.
Better yet, just timestamp the document in the blockchain.
In the US you get a year after public disclosure to file even a provisional. But not so if you want non-US patents. So you are closing some doors but keeping others open with public disclosure. Also, you are starting a 1 year clock.
A provisional is private; it does not count as public disclosure. A provisional is nothing more than a priority date, assuming what you have disclosed in the provisional itself is sufficient, and novel. You can even refile the same provisional every year as long as you believe the subject mater is still novel, but you get a new filing/priority date each time.
As for preventing a 3rd party from patenting the subject matter, either one is sufficient. However, if you publically disclose, only then you also get protection from a 3rd party who builds on your work. So in that case public disclosure is better than the 'secret' provisional.
Better yet, just timestamp the document in the blockchain.
The problem with blogs is that 15 years later it's hard to prove what was really there and when. But having it published on IEEE is the gold-standard.
Chances are that before you get your thing published on IEEE, someone else will start and finish writing an application and maybe even get the patent granted...
[deleted]
It is not enough to succeed; others must fail.
Clickbait.
This is a patent on a very particular form of translation model that handles rare words, i.e. this paper that all the authors are on: http://arxiv.org/abs/1410.8206
This is a patent on a very particular form of translation model that handles rare words, i.e. this paper that all the authors are on: http://arxiv.org/abs/1410.8206
What you wrote is literally the exact meaning I intended with the title, but now reading some comments I see many people are interpreting it as "Google atttempts to patent deep neural networkS for machine translation", (notice the plural) which has a vastly different meaning. I guess I assumed that this audience would know that it wouldn't be remotely possible to patent a broad technique that's already so widely published.
I think that even though the application is for a specific architecture, that this is still worth knowing about, since LSTM is a such well known technique for dealing with sequential data like sentences, and since so far virtually all progress in AI and ML has been driven by academia and has remained open.
I also think it's important to note that pretty much any interesting machine translation task will contain rare words, so even though they've framed it in terms of a specific task, it's one that's actually extremely broad. Machine translation is not so hard when you have tons of data and when you've seen every word many times in combination with its translations. The only really interesting case is when we have to use background knowledge and context to infer meaning. Since natural languages are notoriously ambiguous, this happens all the time. So this app may be broader than it first appears.
I think that even though the application is for a specific architecture, that this is still worth knowing about, since LSTM is a such well known technique for dealing with sequential data like sentences, and since so far virtually all progress in AI and ML has been driven by academia and has remained open.
I also think it's important to note that pretty much any interesting machine translation task will contain rare words, so even though they've framed it in terms of a specific task, it's one that's actually extremely broad. Machine translation is not so hard when you have tons of data and when you've seen every word many times in combination with its translations. The only really interesting case is when we have to use background knowledge and context to infer meaning. Since natural languages are notoriously ambiguous, this happens all the time. So this app may be broader than it first appears.
Doesn't this very paper invalidate this patent, since its publication precedes the filing, and makes the method permanently unpatentable ?
there is a one year grace period after public disclosure to actually file for a patent:
http://www.uspto.gov/web/offices/pac/mpep/s2133.html
http://www.uspto.gov/web/offices/pac/mpep/s2133.html
Also the patent application claims priority to a provisional patent application filed October 24, 2014, which was in all likelihood filed in view of the paper being submitted on October 30, 2014.
Okay, that would allow it - the gap is slightly (1 week) less than a year, and the paper is published by Google employees who presumably are in the patent; the publication would still disallow someone else from patenting it even within that one year.
If memory serves, Google did this with Word2Vec as well, which is a bit sketchy.
If you open source something, there should be a reasonable expectation that it is contributed to the commons. Otherwise a lot of people will build off your stuff, which you can then turn around, file a patent, and claim infringement.
Not a lawyer, but seems like a problem.
If you open source something, there should be a reasonable expectation that it is contributed to the commons. Otherwise a lot of people will build off your stuff, which you can then turn around, file a patent, and claim infringement.
Not a lawyer, but seems like a problem.
Google is patenting this so that if they get sued, they already have a patent. If they actually sue anyone over this FIRST, I'd be shocked. This has got to be a defensive patent, in case someone else was about to patent the same thing and run after them with a lawyer.
Hard to see how it's not another pure-software patent.
They include a function to identify and map source/target unknown ('rare') words.
Oh, and I was wondering what exactly they use in Google Translate.
:(
they will have to pay royalties to Sepp Hochreiter, the inventor of LSTM (http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=679596...)
Why? LSTM as such isn't patented, so the concept is free to use by anyone without any restrictions or royalties, no matter who invented it.
Also, the concept of LSTM wouldn't be patentable directly even in the current liberal software patent interpretation, you might patent particular applications of LSTM (e.g. this patent) but not any and all arbitrary applications of LSTM.
Also, the concept of LSTM wouldn't be patentable directly even in the current liberal software patent interpretation, you might patent particular applications of LSTM (e.g. this patent) but not any and all arbitrary applications of LSTM.
I'm not a patent attorney or anything, but...how could this ever be enforceable...?
You wait until sometime violates the patent, then threaten to sue them. If they don't settle or license, then you actually sue them. If the court decides in your favor you can get damages and they have to stop making the infringing thing. Same as any other patent really!
Why wouldn't this be enforceable? It's just as enforceable as any other software patents - we can (and should!) argue about legality of software patents as such, but if the patent is granted, then you simply take a look at any competing MT system, and if they seem to use this approach, you'll be able to sue them for patent violation.
i'm sure the cost for trying (bit of cash) is greatly outweighed by the benefits of successfully being awarded the patent
And the cost savings of avoiding litigation if a patent troll patents it first.
What are the benefits if you can't enforce it?
Have you been following patent suits in the U.S. at all? You bet they'll be able to enforce it on somebody if they get it. The results pay off the patent and then some.
To defend yourself against patent trolls
Nope. If Google being sued by someone else with a similar business, having patents of their own might help with a countersuit -- Google would allege that PlaintiffX was infringing Google's patents in the course of its own business. But a pure patent troll doesn't operate any business at all (other than collecting license fees), so they can't infringe somebody else's patents in the course of doing the nothing that they do.
[deleted]
Why is Google patenting this tech? It's cutting-edge machine translation. The rest of the industry is still talking about statistical machine translation. The sector is moving very fast. With the recent SyntaxNet release and neural machine translation, Google is approaching the dream: universal automatic translation, which, by the way, is an innovation goal of Obama's administration.
Quick, someone patent using artificial intelligence to generate patentable ideas!
I can't affirm I know about patents, or why some company files a patent. I understand one or two things about translation technology though, and I'm trying to express why I think this is valuable for Google in the context of industrial translation.
Independent claim #1 does not mention neural networks at all. So its an application to patent more broadly than the title suggests.
Neural network(s) is mentioned 55 times in the claims.
That is simply false. Independent claim #1 does mention neural networks.
Aaannd.. I was incorrect, claim #1 does mention neural networks.
(didn't dig too far into this but..) why y'all need to patent this then?
Patents can be beneficial to facilitate constructive competition, but think humanity is best served by neural nets becoming the new electricity rather than the new Apple-esque walled-garden...