Be your own video hoster with Docker and Nginx(datarhei.github.io)
datarhei.github.io
Be your own video hoster with Docker and Nginx
https://datarhei.github.io/restreamer/
31 comments
True, but you can use this to pipe your video source to a CDN for wide distribution.
EDIT: If you're a CDN, it'd behoove you to extend this project to allow for one-click endpoint configuration. Just add API key/secret.
EDIT: If you're a CDN, it'd behoove you to extend this project to allow for one-click endpoint configuration. Just add API key/secret.
I'm streaming from my home plexn box over my internet line even at 1080p the limit is more often than not the internet I have on my portable device at the moment.
Within Europe 1080p is mostly manageable sub 720 is doable even in Asia.
In have 40ish mbit upload capacity.
Never had complaints from my ISP.
Within Europe 1080p is mostly manageable sub 720 is doable even in Asia.
In have 40ish mbit upload capacity.
Never had complaints from my ISP.
40Mbit upload is common?
On residential fttc connections these days I would assume so.
It's not.
I live in Germany. I think we both can agree that I live in, if not the most technically advanced country in Europe that people claim to have very high-quality living standards.
We have about 81.2M citizens and 12.5M can use FTTC / Bandwith plans with upload speeds of 10Mbit+
So about 15.39% of the population of Europes technically most advanced country have access to FTTC.
To see how bad it is, I uploaded you a picture. The colored marks are cities / parts of the country where citizens can get a connection with the quality you claim as "common."
http://i.imgur.com/qsAeRez.png
Also, keep in mind that those kinds of plans are expensive. A plan with 50Mbits upload costs about 40€ per month.
About 4.3M (5%) of the German population live from the german unemployment benefits called "ALG II / Hartz 4".
That is 404€ a month for food, electricity, clothing, and everything else you need to live from.
Which means for 4.3 Million people they'd need to spend 10% of their monthly income on an FTTC connection.
You can see even over here in Europe 40Mbits upload is not quite common if almost 85% of the population don't have access to it and 5% couldn't even afford it if they wanted to.
I live in Germany. I think we both can agree that I live in, if not the most technically advanced country in Europe that people claim to have very high-quality living standards.
We have about 81.2M citizens and 12.5M can use FTTC / Bandwith plans with upload speeds of 10Mbit+
So about 15.39% of the population of Europes technically most advanced country have access to FTTC.
To see how bad it is, I uploaded you a picture. The colored marks are cities / parts of the country where citizens can get a connection with the quality you claim as "common."
http://i.imgur.com/qsAeRez.png
Also, keep in mind that those kinds of plans are expensive. A plan with 50Mbits upload costs about 40€ per month.
About 4.3M (5%) of the German population live from the german unemployment benefits called "ALG II / Hartz 4".
That is 404€ a month for food, electricity, clothing, and everything else you need to live from.
Which means for 4.3 Million people they'd need to spend 10% of their monthly income on an FTTC connection.
You can see even over here in Europe 40Mbits upload is not quite common if almost 85% of the population don't have access to it and 5% couldn't even afford it if they wanted to.
>if not the most technically advanced country in Europe that people claim to have very high-quality living standards
Germany shares it's place with the UK at the bottom of high speed internet availability in Europe (I actually live in the UK but where i live OpenReach (I'm leasing 2 OR truncated lines through IDnet), Virgin Cable are available at their max speeds and very soon 1gig from hyperoptic will too).
FTTC is available throughout most of the UK http://maps.thinkbroadband.com/#!lat=50.50176004214942&lng=6... many other EU countries are far far better off than the UK and apparently Germany.
You only need between 5 and 10mbit upload to being able to stream 1080p, neflix streams 1080p between 4000-5300 kbits based on the profile, Plex has similar profiles depending on if you want to use transcoding or not.
>You can see even over here in Europe 40Mbits upload is not quite common if almost 85% of the population don't have access to it and 5% couldn't even afford it if they wanted to.
That's not exactly the case http://www.ispreview.co.uk/index.php/2015/10/mixed-uk-result... (these are peak figures so they account for network congestion) most of european FTTx subscribers is getting quite close to 40mbit in some countries, but as I said even 10mbit upload should be sufficient for this purpose. Sure I don't expect people living in remote areas with DSL to be able to stream 1080p but that's not really a surprise.
As for the cost, what do the cost have anything to do with it? people who live on unemployment cannot afford many other things, and if you are not employed and living on social benefits perhaps deploying your own streaming service from home should not be on top of your priorities as the computing power, storage, and multiple client devices that are required for such service will or should be out of your reach in the first place.
Germany shares it's place with the UK at the bottom of high speed internet availability in Europe (I actually live in the UK but where i live OpenReach (I'm leasing 2 OR truncated lines through IDnet), Virgin Cable are available at their max speeds and very soon 1gig from hyperoptic will too).
FTTC is available throughout most of the UK http://maps.thinkbroadband.com/#!lat=50.50176004214942&lng=6... many other EU countries are far far better off than the UK and apparently Germany.
You only need between 5 and 10mbit upload to being able to stream 1080p, neflix streams 1080p between 4000-5300 kbits based on the profile, Plex has similar profiles depending on if you want to use transcoding or not.
>You can see even over here in Europe 40Mbits upload is not quite common if almost 85% of the population don't have access to it and 5% couldn't even afford it if they wanted to.
That's not exactly the case http://www.ispreview.co.uk/index.php/2015/10/mixed-uk-result... (these are peak figures so they account for network congestion) most of european FTTx subscribers is getting quite close to 40mbit in some countries, but as I said even 10mbit upload should be sufficient for this purpose. Sure I don't expect people living in remote areas with DSL to be able to stream 1080p but that's not really a surprise.
As for the cost, what do the cost have anything to do with it? people who live on unemployment cannot afford many other things, and if you are not employed and living on social benefits perhaps deploying your own streaming service from home should not be on top of your priorities as the computing power, storage, and multiple client devices that are required for such service will or should be out of your reach in the first place.
Many sub-pewdipie youtube celebs have expressed a desire to host their own content. They want to break free of youtube, but the bandwidth and legal hassles are too much. A turnkey setup read to go on AWS or azure would probably suit them. This isn't that.
By the looks of it, it actually IS a fully featured live video stack including "your own video hoster". But sure, if you want netflix scale, a Raspery Pi on ADSL wont do, at least not for now.
There's a typo on those links in the FAQ (restreamer is repeated):
https://datarhei.github.io/restreamer/restreamer/wiki/dynami...
should be
https://datarhei.github.io/restreamer/wiki/dynamicdns_en.htm...
and
https://datarhei.github.io/restreamer/restreamer/wiki/guide-...
should be
https://datarhei.github.io/restreamer/wiki/guide-buy-hardwar...
(but on that page, all links point to a 404).
https://datarhei.github.io/restreamer/restreamer/wiki/dynami...
should be
https://datarhei.github.io/restreamer/wiki/dynamicdns_en.htm...
and
https://datarhei.github.io/restreamer/restreamer/wiki/guide-...
should be
https://datarhei.github.io/restreamer/wiki/guide-buy-hardwar...
(but on that page, all links point to a 404).
[deleted]
Does anyone know if the nginx+Netflix sendfile changes in Freebsd can be ported to Linux at some point?
Awesome!
Why should one put anything under Docker for production?
Perhaps, chrooted environment is required for effeciently serving static files, I don't know.
And of all apps - nginx, which has been designed to be self-contained (process-isolation is good enough) and has almost zero external dependencies (openssl and pcre, which are parts of any modern OS). I could understand the idea to "containerize" that pile of Java crap with its 100500 maven dependencies just in order not to touch it again, but nginx...
BTW, does Docker's "port mapping", which is, of course, has zero overhead, is what one needs in front of nginx in production?
And of all apps - nginx, which has been designed to be self-contained (process-isolation is good enough) and has almost zero external dependencies (openssl and pcre, which are parts of any modern OS). I could understand the idea to "containerize" that pile of Java crap with its 100500 maven dependencies just in order not to touch it again, but nginx...
BTW, does Docker's "port mapping", which is, of course, has zero overhead, is what one needs in front of nginx in production?
Nginx is only one piece of the puzzle for this app, check out the Dockerfile[1].
[1] https://github.com/datarhei/restreamer/blob/master/Dockerfil...
[1] https://github.com/datarhei/restreamer/blob/master/Dockerfil...
Hej, What is the revenue model ?
Perhaps its just a cool project not trying to make money?
Does docker run on FreeBSD? Because it is crazy to run NGINX on anything other than FreeBSD because of Linux's appalling bad async IO support. For all I know Windows might be a better choice.
Aside from the thousands of applications using Nginx on Linux with fantastic results. Yes, yes we know *BSD has a better networking stack than Linux. Now that you're done letting everyone know how smart you are, please sit down.
> Aside from the thousands of applications using Nginx on Linux with fantastic results.
Citation needed. Name a big company using nginx on Linux to serve up lots of static content.
> Yes, yes we know *BSD has a better networking stack than Linux.
Who mentioned anything about a network stack?
> Now that you're done letting everyone know how smart you are
Well given some people like you clearly don't understand the point I was making I think justifies the comment in the first place.
Citation needed. Name a big company using nginx on Linux to serve up lots of static content.
> Yes, yes we know *BSD has a better networking stack than Linux.
Who mentioned anything about a network stack?
> Now that you're done letting everyone know how smart you are
Well given some people like you clearly don't understand the point I was making I think justifies the comment in the first place.
Citation needed. Name a big company using nginx on Linux to serve up lots of static content.
Facebook[1], Reddit[2], Wordpress[3]
[1] http://venturebeat.com/2012/06/18/nginx-the-web-server-tech-...
[2] https://news.ycombinator.com/item?id=8244868
[3] http://searchdns.netcraft.com/?host=wordpress.com&x=0&y=10
Facebook[1], Reddit[2], Wordpress[3]
[1] http://venturebeat.com/2012/06/18/nginx-the-web-server-tech-...
[2] https://news.ycombinator.com/item?id=8244868
[3] http://searchdns.netcraft.com/?host=wordpress.com&x=0&y=10
Out of interest not challenge, do you have any benchmarks for this at all? I'd be interested in seeing the end results
Well when NGINX introduced thread-pools on Linux to handle async tasks (mainly waiting for block device IO) I think the speed-up was around 8x. Thread-pools make NGINX on Linux tolerable, but it just works flawlessly and beautifully on FreeBSD.
Hey that's not a benchmark! :P
https://www.nginx.com/blog/thread-pools-boost-performance-9x...
...and it was 9x not 8x...
...and it was 9x not 8x...
The 9x delta is from "a synthetic benchmark that simulates the worst mix of blocking and non-blocking operations". Expect real-world workloads to have significantly less performance impact.
The particular issue is: "The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks. That definitely doesn’t make it optimal for many cases.
To solve this problem in particular, thread pools were introduced in NGINX 1.7.11"
So, NGINX doesn't think it's crazy to deploy on linux. And no, Windows is not a better option. "only the select() connection processing method is currently used, so high performance and scalability should not be expected." http://nginx.org/en/docs/windows.html
The particular issue is: "The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks. That definitely doesn’t make it optimal for many cases.
To solve this problem in particular, thread pools were introduced in NGINX 1.7.11"
So, NGINX doesn't think it's crazy to deploy on linux. And no, Windows is not a better option. "only the select() connection processing method is currently used, so high performance and scalability should not be expected." http://nginx.org/en/docs/windows.html
Docker does run of FreeBSD but it heavily relies on ZFS and the 64bit Linux compatibility layer.
https://wiki.freebsd.org/Docker
https://wiki.freebsd.org/Docker
What a bold, unsubstantiated claim.
I heard a while back - not sure where, it could have been on Slashdot but it escapes me - that FreeBSD was dying. Looking into the statistics just now it seems that's the case. FreeBSD is basically dead on the desktop[0] and is dying, no longer even holding a single percent, in the server market.[1] In the High Performance Computing (HPC) market, Linux holds over 99.8% of the top supercomputers.[2]
[0] https://www.netmarketshare.com/operating-system-market-share...
[1] http://w3techs.com/technologies/details/os-freebsd/all/all
[2] http://www.top500.org/statistics/details/osfam/1
[0] https://www.netmarketshare.com/operating-system-market-share...
[1] http://w3techs.com/technologies/details/os-freebsd/all/all
[2] http://www.top500.org/statistics/details/osfam/1
There are millions of Ngnix on Linux production implementations.
And your advice is to use experimental[1] Docker support in FreeBSD, when Nginx would be running inside Docker on top of FreBSD's Linux compatibility layer.
Finally - and most critically - Ngnix introduced thread pools to specifically work around a limitation of Linux's async IO implementation:
Although Linux provides a kind of asynchronous interface for reading files, it has a couple of significant drawbacks. One of them is alignment requirements for file access and buffers, but NGINX handles that well. But the second problem is worse. The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks. That definitely doesn’t make it optimal for many cases.
To solve this problem in particular, thread pools were introduced in NGINX 1.7.11.[2].
[1] https://wiki.freebsd.org/Docker
[2] https://www.nginx.com/blog/thread-pools-boost-performance-9x...
And your advice is to use experimental[1] Docker support in FreeBSD, when Nginx would be running inside Docker on top of FreBSD's Linux compatibility layer.
Finally - and most critically - Ngnix introduced thread pools to specifically work around a limitation of Linux's async IO implementation:
Although Linux provides a kind of asynchronous interface for reading files, it has a couple of significant drawbacks. One of them is alignment requirements for file access and buffers, but NGINX handles that well. But the second problem is worse. The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks. That definitely doesn’t make it optimal for many cases.
To solve this problem in particular, thread pools were introduced in NGINX 1.7.11.[2].
[1] https://wiki.freebsd.org/Docker
[2] https://www.nginx.com/blog/thread-pools-boost-performance-9x...
Being your own video hoster usually means renting a server in a data center with serious bandwidth, or buying streaming service from some cloud provider.[1][2]
[1] http://www.streamingvideoprovider.com/prices.html [2] https://aws.amazon.com/cloudfront/streaming/