OpenAI transcribed over a million hours of YouTube videos to train GPT-4(theverge.com)
theverge.com
OpenAI transcribed over a million hours of YouTube videos to train GPT-4
https://www.theverge.com/2024/4/6/24122915/openai-youtube-transcripts-gpt-4-training-data-google
46 comments
It's a slippery slope. China developed one of the most sophisticated tracking systems to monitor each step be that digital or physical of it's citizens. Should EU/US do the same? IMO no, China is a 'masked' dictatorship and we must not follow every step China does. If they are detected to train their models on global data and we don't want this, simply pass even more laws to block inbound china traffic, apply more sanctions.
You have written "The European Union, for instance, is often viewed as having hindered innovation with excessive regulations over the past decade" - but how do we know if that's true? Maybe its regulation didn't do much and the US growth is simply caused by the printing machine for the global currency combined with an extremely homogeneous market (all ppl speak the same language and globally a lot more ppl speak english compared to say german/french) and it's part of one country compared to EU that's a group of countries operating under some agreements but with still different laws/cultures. How do we know that specifically EU regulation is the root cause of 'less' EU innovation compared to the US and not the other things?
You have written "The European Union, for instance, is often viewed as having hindered innovation with excessive regulations over the past decade" - but how do we know if that's true? Maybe its regulation didn't do much and the US growth is simply caused by the printing machine for the global currency combined with an extremely homogeneous market (all ppl speak the same language and globally a lot more ppl speak english compared to say german/french) and it's part of one country compared to EU that's a group of countries operating under some agreements but with still different laws/cultures. How do we know that specifically EU regulation is the root cause of 'less' EU innovation compared to the US and not the other things?
"China developed one of the most sophisticated tracking systems to monitor each step be that digital or physical of it's citizens. "
that tracking system is not an industrial revolution ...
you are comparing different thing ( that was my mp3 licencing reference , the compromise on preventive legislative micro-management must be evaluated based on importance )
that tracking system is not an industrial revolution ...
you are comparing different thing ( that was my mp3 licencing reference , the compromise on preventive legislative micro-management must be evaluated based on importance )
eu isn't regulating all ai usecases either, just some of them, you instead phrase it like we should let corpos act as they want, unbounded, suck all public data, build stuff that may not be legally allowed in normal cases and make huge profits because that's the only way our civilization advances. We know what corporate greed is, some bounds must be put in place to avoid unintended consequences
a further point could be cryptocurrencies,
they are much more regulated in China than in Europe and the USA,
states choose what to regulate based on the need for control/growth/safeguarding/... everything affects different points differently,
make comparisons with others laws or regulations on other topics must take into consideration the implications that these different categories have on the compared subject
states choose what to regulate based on the need for control/growth/safeguarding/... everything affects different points differently,
make comparisons with others laws or regulations on other topics must take into consideration the implications that these different categories have on the compared subject
"simply pass even more laws to block inbound china traffic, apply more sanctions."
.. like an eu/us "firewall"?
so "they" can have their AI that makes new molecules, new medicines, new materials, increases productivity by 5x , in the meantime should we separate all the scientific research papers because otherwise they will devour us? , so we move backwards but are we giving $5 more to everyone who wrote a book that no one has read? so they can buy the chinese vpn to access their ai to write the next book for the next 5$ ?
so "they" can have their AI that makes new molecules, new medicines, new materials, increases productivity by 5x , in the meantime should we separate all the scientific research papers because otherwise they will devour us? , so we move backwards but are we giving $5 more to everyone who wrote a book that no one has read? so they can buy the chinese vpn to access their ai to write the next book for the next 5$ ?
to remain on t he topic... obv. those 5$ would also go to all youtube channels gosh, those could also be 30$ if you really want to split revenue more equally, so everyone could be 30$ richer and open ai and all newcomers under that legislation could be dead, yeah probably we'll specialize on niche , small, and costly ai each with their limited and licensed dataset
yes, they will have own models, we will have own models (if the law permits, since not every usecase should be allowed). Even the productivity increase by 5x must be approached carefully, like what if it causes massive unemployment (since ppl working on automated jobs will suddenly be out of working field and will not be able to adapt/switch to other areas fast enough and be homeless as result). Some regulations are/can be bad, but other must be put in place to avoid unwanted consequences. The unbounded capitalism at all cost is a thing in US but not in EU and even less in China.
What strikes me, why do you think that EU/US can't do all this stuff if they put in place some regulations (like molecules,medicines,materials, etc...)? Regulations (normally) should limit only a subset of usecases while allowing others, don't you think it's a normal approach instead of letting big corpos suck for free any public data(be it copyrighted or personal) and act unbounded while profiting from this? We know what corporate greed is, look at the insulin price in US vs EU, why don't you think that the same will happen with AI tools/corpos?
What strikes me, why do you think that EU/US can't do all this stuff if they put in place some regulations (like molecules,medicines,materials, etc...)? Regulations (normally) should limit only a subset of usecases while allowing others, don't you think it's a normal approach instead of letting big corpos suck for free any public data(be it copyrighted or personal) and act unbounded while profiting from this? We know what corporate greed is, look at the insulin price in US vs EU, why don't you think that the same will happen with AI tools/corpos?
> Moreover, this discussion isn't just about licensing MP3s; it's about managing common knowledge—that which is accessible to everyone, including competitors in different jurisdictions—for the creation of an industrial revolution.
Great, why don't these companies train their models on that "common knowledge" then and leave copyrighted work alone? Would be a great option wouldn't it? Solves the whole problem in one go.
Personally, I think these companies are just stealing copyrighted works, and should be sued for that. It's against the law, it's pretty simple.
And the whole China argument... Sorry but it makes no sense. They could beat us in AI by doing illegal things, so we should do the same illegal things to not let them win?
China has a lot of slave labour too. By the same reasoning, shouldn't we introduce slavery to not let them win production?
Great, why don't these companies train their models on that "common knowledge" then and leave copyrighted work alone? Would be a great option wouldn't it? Solves the whole problem in one go.
Personally, I think these companies are just stealing copyrighted works, and should be sued for that. It's against the law, it's pretty simple.
And the whole China argument... Sorry but it makes no sense. They could beat us in AI by doing illegal things, so we should do the same illegal things to not let them win?
China has a lot of slave labour too. By the same reasoning, shouldn't we introduce slavery to not let them win production?
I'm not saying that in the future there won't be licensing methods that perhaps will serve to provide more direct access to the data, for example free = you can find them online, or do scraping... if you pay for them you have direct access to the dataset for training in real time , and you know that it is clean, without "watermarks" (even textual), etc..
the comparison with slave labor has nothing to do with it because here you are not violating a person's body to train AI, it is more like a duplication (but actually reinterpreted) of digital information that can be copied infinite times without damage, if I read it 3 times a PDF protected by copyright does not cause 3x damage compared to just once
I see it as more similar, for example, to how when the first search engines were born... if they had had to ask each site for permission to read, rework and provide the "external" search service, everything would have died immediately (at least in Europe and use )
another example is allowing emerging countries to maintain lighter copyright laws to facilitate their growth (yes, they violate them, yes perhaps a small percentage of Zambian inhabitants would perhaps buy the media by paying Western fees, but it is better to leave it alone and look at a greater good than managing copyright guarantees everywhere...) the Americans interpret this legislative concept better, the Europeans if they don't wake up will be left behind for a long time,
and I repeat, comparing the passing of a dataset containing copyrighted material into matrices for the generation of an AI is very different from slavery
I see it as more similar, for example, to how when the first search engines were born... if they had had to ask each site for permission to read, rework and provide the "external" search service, everything would have died immediately (at least in Europe and use )
another example is allowing emerging countries to maintain lighter copyright laws to facilitate their growth (yes, they violate them, yes perhaps a small percentage of Zambian inhabitants would perhaps buy the media by paying Western fees, but it is better to leave it alone and look at a greater good than managing copyright guarantees everywhere...) the Americans interpret this legislative concept better, the Europeans if they don't wake up will be left behind for a long time,
and I repeat, comparing the passing of a dataset containing copyrighted material into matrices for the generation of an AI is very different from slavery
> Personally, I think these companies are just stealing copyrighted works, and should be sued for that. It's against the law, it's pretty simple.
Is it copyright infringement to count how many times each letter appears in a book?
I don't know that it is, and if it's not, then there is at least some line you can draw where mechanically reading and learning from a copyrighted work is not copyright infringement.
The question of whether training a transformer model is on the legal side of that line remains to be seen, but I don't think it's as clear cut as you make it out to be.
Is it copyright infringement to count how many times each letter appears in a book?
I don't know that it is, and if it's not, then there is at least some line you can draw where mechanically reading and learning from a copyrighted work is not copyright infringement.
The question of whether training a transformer model is on the legal side of that line remains to be seen, but I don't think it's as clear cut as you make it out to be.
From the NY Times article:
> As long as you can get over the synthetic data event horizon, where the model is smart enough to make good synthetic data, everything will be fine,” Mr. Altman said.
This seems intuitively incorrect to me. How can you advance the capabilities of a model by using the output of a model of the same level?
> As long as you can get over the synthetic data event horizon, where the model is smart enough to make good synthetic data, everything will be fine,” Mr. Altman said.
This seems intuitively incorrect to me. How can you advance the capabilities of a model by using the output of a model of the same level?
How can AlphaGo advance capabilities by using the output of a model of the same level? It's not as simple in this domain but there are ways.
Go can be modeled with perfect accuracy. The feedbackloop is perfect.
LLMs have no feedback loop with reality if it is short circuited with itself. It needs fresh training data.
LLMs have no feedback loop with reality if it is short circuited with itself. It needs fresh training data.
I think that’s different. AlphaGo is using reinforcement learning in a context in which there is a clear evaluation function— did a strategy lead to a win or loss.
I said it's not as simple but there are ways - e.g. you can generate more of your best quality of data, you can try to model the direction of quality or an objective, you can have minor human input at some points to judge which direction performs better, you can objectively verify some of your input to use as a partial objective - code, math, logic etc.
Which million hours though? The quality of training material appears to be the limiting bus factor for all LLMs. Maybe the future will be quality training materials from subject matter experts run in a DRM'ed environment. I think that's probably one of the ways people with specialized knowledge will make money in the future, but they should also get "residuals" based on end-user AI use.
Fascinating to think how good GPT4 is considering this is < 1% of available YouTube data. Lots of opportunities to improve the model. Future is bright.
It's probably about curating quality content. Most of YouTube is dross. How did they they find a million plus YouTube videos?
StolenAI
GPT-2: Reddit [0]
GPT-3.5: ?
GPT-4: YouTube
GPT-5: ???
[0]https://www.youtube.com/watch?v=13CZPWmke6A&t=3645s
GPT-3.5: ?
GPT-4: YouTube
GPT-5: ???
[0]https://www.youtube.com/watch?v=13CZPWmke6A&t=3645s
>GPT-5: ???
Perhaps capture of audio and video from places lacking legal expectation of privacy.
Buying conversations from listening devices at live nation sports venues or clear channel airports.
Spooky, but probable.
Maybe there are troves of data yet to be unearthed, or needing extraction from legacy media formats. Orgs aren't desperate enough to do that kind of schlep, but soon enough.
Like archives of terrestrial AM/FM or even HAM radio up for grabs. Going backward to go forward.
Perhaps capture of audio and video from places lacking legal expectation of privacy.
Buying conversations from listening devices at live nation sports venues or clear channel airports.
Spooky, but probable.
Maybe there are troves of data yet to be unearthed, or needing extraction from legacy media formats. Orgs aren't desperate enough to do that kind of schlep, but soon enough.
Like archives of terrestrial AM/FM or even HAM radio up for grabs. Going backward to go forward.
GPT-5: Classified Information
GPT-5: Microsoft Office 360 customer data (Word, Excel, PowerPoint, SharePoint, Microsoft Dynamics)
Microsoft says they don't train on Office 365 customer data. Maybe you don't believe them, but that's what they say.
According to Mozzila four lawyers, three privacy experts, and two campaigners looked at Microsoft's new Service Agreement, and none could tell if Microsoft plans on using your personal data...
https://foundation.mozilla.org/en/campaigns/microsoft-ai/
https://news.ycombinator.com/item?id=37347528
https://foundation.mozilla.org/en/campaigns/microsoft-ai/
https://news.ycombinator.com/item?id=37347528
Teams is slow for a reason. /s, but only slightly.
spotify and xvideos
Podcasts will be next
Related: Arvid Kahl (who hosts the podcast The Bootstrapped Founder) recently launched https://podcast.fm , which essentially transcribes thousands of podcasts and makes them available for search.
They already did that. The main reason why they created Whisper is to transcribe and train on podcast data
Yes, AI/LLM audio generation is going to be huge. A lot of free podcasts are already distributed as YouTube videos with the only "video" part being the host speaking into the microphone. Some podcasting companies like Acast are sitting on a gold mine of AI training content.
https://www.acast.com/
https://www.acast.com/
Abusive to think of peoples creative ip as a gold mine to exploit just because we live in a world where companies can change their tos at will, so everyone signs blank contracts to live daily life that say whatever the other party ever wants
Hold on. I never said that the content creators shouldn't get their fair share. But there is clearly a role for aggregators to play here as well.
As a random citizen that isn’t involved in these companies, I have no problem with OpenAI doing this anymore than I would with Google doing it. I don’t see why Google should have exclusive rights to use content that others have made just because they’re protected from competition (for YouTube) via network effects and anti-competitive practices. I’m okay with the actual video creators exercising rights and controls over the use of their content, but I would argue that Google should itself not just assume rights to train on YouTube data through expansive or updated terms of service. And it seems like that’s exactly what they did - quietly assume rights to train their AI models on YouTube content without any additional compensation to content creators:
> Google also gathered transcripts from YouTube, according to the Times’ sources. Bryant said that the company has trained its models “on some YouTube content, in accordance with our agreements with YouTube creators.”
> The Times writes that Google’s legal department asked the company’s privacy team to tweak its policy language to expand what it could do with consumer data, such as its office tools like Google Docs. The new policy was reportedly intentionally released on July 1st to take advantage of the distraction of the Independence Day holiday weekend.
There needs to be legislation to reign in the practices of big tech and also this increasingly common dark pattern of forcing updated terms on customers. They can get away with this due to a lack of feasible competition and their massive capital. Only regulation or taxation might resolve this issue so that control lies with content creators.
> Google also gathered transcripts from YouTube, according to the Times’ sources. Bryant said that the company has trained its models “on some YouTube content, in accordance with our agreements with YouTube creators.”
> The Times writes that Google’s legal department asked the company’s privacy team to tweak its policy language to expand what it could do with consumer data, such as its office tools like Google Docs. The new policy was reportedly intentionally released on July 1st to take advantage of the distraction of the Independence Day holiday weekend.
There needs to be legislation to reign in the practices of big tech and also this increasingly common dark pattern of forcing updated terms on customers. They can get away with this due to a lack of feasible competition and their massive capital. Only regulation or taxation might resolve this issue so that control lies with content creators.
> I don’t see why Google should have exclusive rights to use content that others have made
Two perspectives: (1) because google is paying for the storage, processing and delivery of these videos, so it makes sense that they could do something with them that others can. And (2) because maybe you uploaded those video to YouTube specifically because you find Sundar Pichai vision extremely interesting so you definitely want to be on his platform and you find Sam Altman so obnoxious so you don't want him to use your content.
Two perspectives: (1) because google is paying for the storage, processing and delivery of these videos, so it makes sense that they could do something with them that others can. And (2) because maybe you uploaded those video to YouTube specifically because you find Sundar Pichai vision extremely interesting so you definitely want to be on his platform and you find Sam Altman so obnoxious so you don't want him to use your content.
That isn't a whole lot in the grand scheme of things, wonder how they picked the videos. YouTube gets like 700k hours of videos uploaded a day, and a million hours is probably like 10bn tokens out of the 13tn for GPT4
I concur
english speaking rate: ~150 words per minute
gpt tokens per english word: ~1.3 tokens per word
1M hours = ~12B tokens
I'd fudge that up a bit because it's probably more tokens from non-English but comparatively seems like not much
english speaking rate: ~150 words per minute
gpt tokens per english word: ~1.3 tokens per word
1M hours = ~12B tokens
I'd fudge that up a bit because it's probably more tokens from non-English but comparatively seems like not much
I think the fair use argument levied is fine. They are talking about using Whisper to transcribe videos and then train the model on that. Both fair use and derivative works are satisfied there. That fulfills the actual law.
The terms of service though? who cares? deactivate their google account then, they can still access the videos unauthenticated.
Personally I’ve made so much money off of AI generated works that would not be protected by copyright at all, that I think creators and owners of copyrighted works should re-evaluate their relationship with copyright. It seems more like copyright gives a sense of pride and accomplishment while almost no work is being monetized by its copyright or license alone.
The terms of service though? who cares? deactivate their google account then, they can still access the videos unauthenticated.
Personally I’ve made so much money off of AI generated works that would not be protected by copyright at all, that I think creators and owners of copyrighted works should re-evaluate their relationship with copyright. It seems more like copyright gives a sense of pride and accomplishment while almost no work is being monetized by its copyright or license alone.
Just curious, how much money did you make before AI with content that would be protected by copyright? Wondering if you’re someone who was on one side of the fence and it’s now on the other.
hmm I’ve made a lot there too
but I would say worrying about protecting that right and “doing it right” is more work than actually having a license agreement or court filing, both of which are rare
It is liberating to be told “this can’t have copyright” and make enough money from entertaining the humans, who cares if it gets replicated and why didnt they just AI generate their own thing
but I would say worrying about protecting that right and “doing it right” is more work than actually having a license agreement or court filing, both of which are rare
It is liberating to be told “this can’t have copyright” and make enough money from entertaining the humans, who cares if it gets replicated and why didnt they just AI generate their own thing
I don't think people should be using videos without permission. But that said, I think we should err on the side of leniency, because if we allow openai to use the data, we will allow their competitors to as well.
If push comes to shove, openai can acquire all the data they need with microsoft money.
But the little guys can't. So if we let them get away with it, we are really making sure the little guys can compete!
If push comes to shove, openai can acquire all the data they need with microsoft money.
But the little guys can't. So if we let them get away with it, we are really making sure the little guys can compete!
If we want "the little guys" to have our data we can give it to them, it's not as if their only option is to steal it.
If it’s available on the open internet for anyone to consume, I don’t see a problem.
If the USA and Europe take this route, what happens next? It's conceivable that China might not follow suit, opting instead to leverage all available online data to boost its AI development. This could lead to a scenario where China manages to develop a super AI, while Europe and the USA get caught in a complex maze of licenses and expensive, specialized micro-AIs.
Faced with an advanced and accessible Chinese AI, what will users around the world do? It's plausible that despite initial concerns, many will end up using the Chinese AI, thereby giving it even more power, data, and growth capacity.
This brings us to the importance of balanced regulation. The European Union, for instance, is often viewed as having hindered innovation with excessive regulations over the past decade. If the USA were to follow this path, they might risk limiting their own capacity for innovation.
Excessive regulation thus emerges as a double-edged sword: on one hand, it aims to protect privacy and user rights; on the other hand, it risks stifling innovation and making one's companies and technologies less competitive globally. In this scenario, the question becomes: How can the USA and Europe balance data protection needs with the necessity to remain competitive in the global AI arena?
Moreover, this discussion isn't just about licensing MP3s; it's about managing common knowledge—that which is accessible to everyone, including competitors in different jurisdictions—for the creation of an industrial revolution. Before clamping down with regulations, careful consideration is needed. Licensing mechanisms can be introduced later if necessary, but they should not provide an undue advantage to competitors. Balance is essential.