Categories
4k IT Servers Streaming

Another Nginx Re-Streamer With HLS Output

Recently I had a request to see if I would be able to provide some sort of a tutorial on how to make a re-streaming server for Facebook and YouTube as well as having an HLS output as well, preferably with various resolutions.

My server is based on the rtmp-module in nginx. We should install this on an Ubuntu server from Lightsail. I usually make a very small one for $3.50 per month, but then if I want to stream with HLS I need to use a snapshot and create a new instance that is much larger. Otherwise it won’t be able to handle all the transformations of the data to make the HLS files and it will fail.

This server in the stock configuration as shown will re-stream to any other RTMP locations that you need and also send out HLS to places in various resolutions. If you send it a 1080p 30fps input, it will send out a 1080, 720, and 360 HLS output as well. If you make some adjustments that I show in an optional section, then you can make it take in 4k and re-stream that to any RTMP places you wish, plus output on HLS 4k, 1080, 720, and 360.

Install nginx and update ubuntu

sudo apt update
sudo apt upgrade
sudo apt install -y nginx
sudo apt install -y libnginx-mod-rtmp

Now we have to install a bunch of needed packages to allow for the media streaming to properly occur.

sudo apt install -y software-properties-common
sudo dpkg --add-architecture i386
sudo apt update

The entire below code should be copied and pasted in its entirety at once.

sudo apt install wget nano python-certbot-nginx ufw unzip software-properties-common dpkg-dev git make gcc automake build-essential joe ntp ntpdate zlib1g-dev libpcre3 libpcre3-dev libssl-dev libxslt1-dev libxml2-dev libgd-dev libgeoip-dev libgoogle-perftools-dev libperl-dev pkg-config autotools-dev gpac ffmpeg sysstat nasm yasm mediainfo mencoder lame libvorbisenc2 libvorbisfile3 libx264-dev libvo-aacenc-dev libmp3lame-dev libopus-dev libfdk-aac-dev libavcodec-dev libavformat-dev libavutil-dev g++ libc6:i386 freeglut3-dev libx11-dev libxmu-dev libxi-dev libglu1-mesa libglu1-mesa-dev
sudo apt install mariadb-server mariadb-client phpmyadmin php php-cgi php-common php-pear php-mbstring php-fpm

These are to allow for the rtmp module to be installed and the statistics to be made.

cd /usr/src

sudo git clone https://github.com/arut/nginx-rtmp-module

sudo cp /usr/src/nginx-rtmp-module/stat.xsl /var/www/html/stat.xsl

This is the crossdomain file

sudo nano /var/www/html/crossdomain.xml

Paste the below into that file.

<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy SYSTEM "http://www.adobe.com/xml/dtds/cross-domain-policy.dtd">
<cross-domain-policy>
<allow-access-from domain="*"/>
</cross-domain-policy>

This is the info.php file.

sudo nano /var/www/html/info.php

Paste the below into that file.

<?php
phpinfo();
?>

This is to create the locations of the hls streams.

sudo mkdir /var/livestream
sudo mkdir /var/livestream/hls
sudo mkdir /var/livestream/med
sudo mkdir /var/livestream/low

This is to give the proper permissions the livestream location can be used.

sudo chown -R www-data: /var/livestream

Now we have to create the nginx.conf file. This is where the majority of the server gets its programing.

sudo nano /etc/nginx/nginx.conf

The below file should be used as the nginx.conf file. I took off the .conf extension and made it a .txt so the site would allow me to post it.

In all the files that I have here, if you use them, please be sure to change the IP to your IP address. You will have to change them by using “sudo nano” and the file name such as nginx.conf. If you click on the link below that says “nginx”, it will open up a new browser window and you can copy and paste this into your nginx.conf file.


Skip this section if you don’t need 4K.

Below is the nginx.conf file that you should use if you want the streamer to make 4 types of HLS files. It can take 4k and create 4k, 1080, 720 and 360 outputs. But it needs a very large server to do it all. Also you’ll have to create the proper html files that allow you to show people the created outputs, or just use it to send to other services. I didn’t include the proper html files in this tutorial. You should name this “nginx.conf” if you use it.

Remember that you’ll also need to make directories where the “high” files will be stored.


sudo nginx -t
sudo systemctl restart nginx
sudo nano /etc/nginx/sites-available/default

The below default.txt is actually the file that is from /etc/nginx/sites-available/default. I added the .txt extension to it so it would allow me to post it properly. You should delete the .txt extension to make it work if you want to use it directly, or just copy and paste it into the default file when you are in the nano program.

sudo nginx -t
sudo systemctl restart nginx

So that we can create and use a website to manage and check on these streaming server functions we have to have a video player that will play our HLS files on a website. The first two lines below should be copied and pasted individually. The “sudo wget” sections should be done as sections. That will get the .zip file and the .js file that you will need for the video player.

sudo mkdir /var/www/html/videojs
cd /var/www/html/videojs

sudo wget https://github.com/videojs/video.js/releases/download/v7.7.6/video-js-7.7.6.zip

sudo wget https://github.com/videojs/http-streaming/releases/download/v1.13.1/videojs-http-streaming.js

The following lines should be done individually and they will all the video player to be unzipped as well as make the proper allowance for it to be used with the directories we created.

sudo unzip /var/www/html/videojs/video-js-7.7.6.zip
sudo chown -R www-data: /var/www/html
sudo ls -la /var/www/html/videojs

Now we have to make the page that will show the options for the server.

We have to make the index.html page as well as the pages for the videos, low.html, med.html, live.html.

Here are some copies of my files. You’ll have to modify them slightly and add in your IP address on each of them to make them operate. Just replace the IP in some of the lines with your IP.

Note that these are very poor HTML code. I just made them quickly so that I could get a working page for myself. They aren’t neat and they aren’t good. They just work. Maybe sometime in the future I might fix them a bit, but for now I have other things that are more pressing than making them nice and pretty.

sudo nano /var/www/html/index.html
sudo nano /var/www/html/live.html
sudo nano /var/www/html/med.html
sudo nano /var/www/html/low.html

The index file below is set up for making 3 outputs (Low, Med, Live). The Live file output is the same stream output resolution that the input comes into the server. The below is a rough index.html file. It should make a workable webpage where you can choose to see some of your stats or a feed from each of the 3 above resolutions.

<h2 style="text-align: center;"><strong>Streaming Status Pages</strong>&nbsp;</h2> <p style="text-align: 
center;">&nbsp;</p> <p style="text-align: center;"><a href="https://(YOUR IP.Address)/nginx_status" target="_blank" 
rel="noopener">Status of Connections</a></p> <p style="text-align: center;">&nbsp;</p> <p style="text-align: 
center;">&nbsp;</p> <p style="text-align: center;"><a href="https://(YOUR IP.Address)/stat.html" target="_blank" 
rel="noopener">Status of the Stream</a></p> <p>&nbsp;</p> <p style="text-align: center;">&nbsp;</p> <p 
style="text-align: center;">Feed for Computers? <a href="https://(YOUR IP.Address)/live.html" target="_blank" 
rel="noopener">Live Stream "Full 1080 Resolution"</a></p> <p style="text-align: center;">&nbsp;</p> <p 
style="text-align: center;">&nbsp;</p> <p style="text-align: center;">Feed for the App: <a 
href="https://(YOUR IP.Address)/low.html" target="_blank" rel="noopener">Live Stream "Low Resolution"</a></p> <p 
style="text-align: center;">&nbsp;</p> <p style="text-align: center;">&nbsp;</p>

<p style="text-align: center;">Feed for Computers?: <a href="https://(YOUR IP.Address)/med.html" target="_blank" 
rel="noopener">Live Stream "720 Resolution"</a></p> <p style="text-align: center;">&nbsp;</p> <p 
style="text-align: center;">&nbsp;</p> <p style="text-align: center;">&nbsp;</p> <p style="text-align: 
center;">&nbsp;</p> <p style="text-align: center;">&nbsp;</p>
<p>&nbsp;</p>

Now you should test it all by streaming to your RTMP location at rtmp://IP/live/stream. If it takes hold and your streamer attaches then you can click on the websites links to check the feed.

After we have that all running. We have to get stunnel so that facebook will work. We’ll go back to the command prompt from your server. Issue the following commands to get stunnel.

sudo apt-get install stunnel4 -y

Now we’ll have to change stunnel’s boot configuration, issue the following command:

sudo nano /etc/default/stunnel4

Change Enabled from 0 to 1. It should look like the following:

ENABLED=1

Next we have to edit the stunnel configuration file.

sudo nano /etc/stunnel/stunnel.conf

You’ll have to cut and paste this in its entirety. It should look like this:

pid = /var/run/stunnel4/stunnel.pid
output = /var/log/stunnel4/stunnel.log
setuid = stunnel4
setgid = stunnel4
# https://www.stunnel.org/faq.html
socket = r:TCP_NODELAY=1
socket = l:TCP_NODELAY=1
debug = 4
[fb-live]
client = yes
accept = 1936
connect = live-api-s.facebook.com:443
verifyChain = no

Then of course you’ll have to use ctrl-x to exit, and Y to save it as the original named file.

Next we have make it enabled after boot by doing the following:

sudo systemctl enable stunnel4.service

Now we have to restart stunnel because we changed the configuration files.

sudo systemctl restart stunnel4.service

Now since we changed the stunnel configuration and all we should restart NGINX for good measure.

sudo systemctl restart nginx
Categories
IT Servers Streaming

Another Type of Re-Streamer Using Node.js

I came across another type of re-stream server that might help some people. It can also be run on a $3.50 per month server from Amazon Lightsail. It’s a node.js server called Node-Media-Server roughly according to this link: https://github.com/illuspas/Node-Media-Server. I’ve modified it slightly and made it useful for re-streaming to YouTube and Facebook.

The reason I searched out yet another re-streaming server type is that I found that streaming with Wirecast going to my other server under NGINX was unreliable. That server works very well with our hardware streamer and the streaming apps on iPhone and Android but it didn’t work well with Wirecast. The following server works very well with Wirecast as well as our hardware streamer and the apps that I have on my iPhone and Android phone.

My version here, is slightly different than the one on Github and I’ve laid out some instructions below. Please read the entire post before starting as we want to make sure it goes as smoothly as possible.

The first step in the process is to to to Amazon Lightsail. The website address is : https://aws.amazon.com/free/compute/lightsail/ They even have a one month free trial of their services for the smallest one. That’s the one we want so it’s great just to try for this. So even if you don’t know what you’re really doing, you aren’t even out anything to try.

Go to Get Started Now and begin. You’ll have to create an Amazon AWS account to start, but that’s easy. (I bet you’ll like it after the first month too) I can’t go back through this because I already have an account. I don’t think I can walk you through some of those details because of that.

However once you are through with that, you should get to a page that looks like this:

From here you choose your instance location. I always choose Northern Virginia for two reasons. One is that I’m located 40 miles from these server farms. Two is that these are probably some of the fastest server farms in the world.

Next you have to pick the OS for the server. Also you have to choose “OS Only” and I choose Linux with Ubuntu being the image I want. It runs the software amazingly and the instructions assume these choices. Now we have to name our “instance”. I like something like “re-streamer” or something of the sort.

Next you have to wait for it to be created, that takes about a minute or less. Once it’s up and running, we can log into it via the browser. That is very simple and you just click the hyperlinked name of your server. That brings you to a page with a huge button that says “Connect Using SSH”. That’s what we want.

Then it brings you to a command prompt. This is where we’ll issue all our commands to make the server do all our work.

Then you can begin with the following code at the command prompt. Press “y” when it asks and select “yes” by putting the red curser over it when you see a graphic ask a question. Or if it asks to press [enter] please do that.

My directions below are slightly different than the ones on Github but these solve some little issues you might find if you followed the others.

The below commands you should perform one at a time by cutting and pasting into the command prompt.

mkdir nms
cd nms
sudo apt-get install npm
npm install node-media-server
sudo apt install software-properties-common
sudo apt update
sudo add-apt-repository ppa:jonathonf/ffmpeg-4
sudo apt install ffmpeg
sudo nano app.js

When you use “sudo nano app.js” you’ll be opening a file. This is the main configuration file for the node-media-server. You’ll need to paste in the following code but put in any locations you want to send your stream to via RMTP. You have to replace my keys with your keys obviously.

const NodeMediaServer = require('node-media-server');
const config = {
  rtmp: {
    port: 1935,
    chunk_size: 60000,
    gop_cache: true,
    ping: 30,
    ping_timeout: 60
  },
  http: {
    port: 8000,
    allow_origin: '*'
  },

  relay: {
        ffmpeg: "/usr/bin/ffmpeg",
        tasks: [{
                app: "live",
                mode: "push",
                edge:"rtmp://127.0.0.1:1936/rtmp/<<YOUR FB KEY>>",
                appendName: false
                },
                {
                app: "live",
                mode: "push",
              
edge:"rtmp://a.rtmp.youtube.com/live2/<<YOUR YT KEY>>",
                appendName: false
                }
               ]
        }
  };
var nms = new NodeMediaServer(config)
nms.run();

We have to get stunnel so that facebook will work. We’ll go back to the command prompt from your server. Issue the following commands to get stunnel.

sudo apt-get install stunnel4 -y

Now we’ll have to change stunnel’s boot configuration, issue the following command:

sudo nano /etc/default/stunnel4

Change Enabled from 0 to 1. It should look like the following:

ENABLED=1

Next we have to edit the stunnel configuration file.

sudo nano /etc/stunnel/stunnel.conf

You’ll have to cut and paste this in its entirety. It should look like this:

pid = /var/run/stunnel4/stunnel.pid
output = /var/log/stunnel4/stunnel.log
setuid = stunnel4
setgid = stunnel4
# https://www.stunnel.org/faq.html
socket = r:TCP_NODELAY=1
socket = l:TCP_NODELAY=1
debug = 4
[fb-live]
client = yes
accept = 1936
connect = live-api-s.facebook.com:443
verifyChain = no

Then of course you’ll have to use ctrl-x to exit, and Y to save it as the original named file.

Next we have make it enabled after boot by doing the following:

sudo systemctl enable stunnel4.service

Now we have to restart stunnel because we changed the configuration files.

sudo systemctl restart stunnel4.service

Now that Stunnel is installed and the app.js file has been properly created we can try it out. To make it run you have to execute the app.js file. Use the following command. The & after the command will let it run in the background and won’t close when you close the command prompt

node app.js &

This should output something like the following:

Whenever you have to modify anything in that app.js file remember that it’s located in your home folder, under “nms”. So in linux you have to go to that location from login like this if you want to directly:

sudo nano ubuntu/nms/app.js

Then remember that to start it again you have to use the command “node app.js” or “node app.js &”.

Also if you have to stop the app.js from running you can press cntrl-x, which will exit the process and give you a command prompt again.

You’ll want to fix the firewall settings on the lightsail server instance so that our connections from outside can get to the server. You should allow ports, 8000,443, and 1935 along with the ports that are automatically allowed by lightsail. You can change this by going to the “networking” section of the instance. Then making it look like the picture below. You’ll also want to set the IP address of your streaming location so that other people can’t stream to your instance. the instance I created for this tutorial isn’t going to be running by the time you read this. I’ll be deleting it, so it will be safe.

Next you can test it with some simple software like VLC along with OBS or your churches streaming software or hardware encoder. You’ll need your RTMP feed to go to your server IP address which is seen on the home page of the instances of Lightsail. It’s on the bottom right above your server location. You can also set it to make it static at this point, but that will give it a new one that will not change. You’ll want that so you don’t always have to change your OBS or streaming software or hardware. To make a static IP just go to the instance by clicking the hyperlinked name and then choose networking from the options across the middle of the page. Then choose static IP on the left.

This gives a nice output page as well to see the status of your running server that’s waiting for your stream. Go to your IP address 000.000.000.000:8000/admin

In OBS or your hardware encoder choose these options in the stream settings:

Service: Custom
Server: rtmp://<<ngnix server IP address>>/live
Stream Key: test

For a playback test use VLC or something that can play RTMP streams. Set it up as follows: Go to Network Stream, Network.

network URL: rtmp://<<node-media-server IP address>>/live/test

You can also go see what the admin page looks like at that IP address:8000/admin, it will show you the inputs and outputs as well as memory usage and cpu usage.

Also now you should be able to go to Facebook and YouTube to see it there to go live with if you wish. Everything should be up and running at this point. You might want to take a snapshot here, because that way you can always fix anything else that goes wrong. Also if you want to further modify this for other purposes to make it better later you have a good jumping off point to create a new instance without jeopardizing your running one.

******The below section about PM2 is an addition to this post to make the program work better for streaming events without worrying about the terminal window closing or disconnecting.

To make this app.js function continue to work even if you close the command prompt, which you will probably want to do reliably. You should install another small program that will make this continue in the background and continue pushing your streams anytime you need them. We’re going to use PM2 to do this.

Install PM2 by typing the following at the command line:

sudo npm install pm2 -g

There are some huge benefits for you if you run your application using pm2. Make sure you are in the directory /nms/ where you installed the app.js file, and instead of running the app as we did above (with node app.js), we’ll run it now using the following command:

pm2 start app.js

This will show something like:

This PM2 kept our server up and running without any input for a 24 hour test we inadvertently ran yesterday through this morning.

If you have to re-do any of the app.js file you might have to restart the pm2 command, some of the below might be helpful as well in that case. If you have to restart the pm2 command, sometimes it will cause two running processes. You can check what’s running under pm2 processes by using the command pm2 list

pm2 list               # this lists the running processes
pm2 stop all           # Stop all processes
pm2 restart all        # Restart all processes

pm2 reload all         # Will 0s downtime reload (for NETWORKED apps)

pm2 stop 0             # Stop specific process id, the id is shown as a 0 in the examples here
pm2 restart 0          # Restart specific process id

pm2 delete 0           # Will remove process from pm2 list
pm2 delete all         # Will remove all processes from pm2 list

# Misc

pm2 reset <process>    # Reset meta data (restarted time...)
pm2 updatePM2          # Update in memory pm2
pm2 ping               # Ensure pm2 daemon has been launched

Also remember to type “exit” to get out of a terminal window safely. Don’t just close the window with the X on the top.

If you need more help, please email me tim@timstreams.com or find me on Facebook.

Categories
IT Servers Streaming

Virtual Services and Streaming

This is what our server configuration looks like. This is a quick document that I made up to show the leadership what I did.

Because of our new uptick in streaming we had to figure out how to stream our live services with RTMP to Facebook and YouTube, as well as our app. The app is a little odd as it requires a stream input of a .m3u8 file, also called HLS. Our church cable modem is not able to support more than one 1080 x 30fps output at once. This is about a 4mps stream. So we needed something to help us.

This led us to research services like restream.io and Boxcast. Both these will do it for RTMP, however Boxcast will do that and also the .m3u8 (HLS). That brought up an interesting challenge in my mind. I figured that this couldn’t be “that” hard. I just had to figure out how to re-stream RTMP. That led me to cloud servers which I had a tiny bit of experience with after-all this website is on an AWS Lightsail cloud server which costs $3.50 and my other website timrides.com is on a free Google cloud server instance.

I had experience in spinning up cloud servers with Bitnami originally and running apache as a webserver. I had seen the wonders of what can be done, but didn’t know how to do more than run WordPress on those Bitnami installations for the websites.

So I went to Amazon and to the Lightsail section which is sort of the simple part of AWS where you pay for various size cloud servers and all your data and things are included. It makes billing simple and you don’t have to worry much about causing some huge bill. You can get there at https://lightsail.aws.amazon.com/ .

The first part of making the re-streaming server for RTMP is getting an instance such as the $3.50 one, it’s good for at least re-streaming to 3 places. So you take in your RTMP feed from your streaming software like wirecast or obs and you send it to the server. The server takes it and sends it on to the services like YouTube and Facebook. If you need more, you might need a bigger server like the $5 one. In any event its cheap and even if one is to small you can take a snapshot of it and spin up a bigger one in about 4 minutes. I chose to run Ubuntu 18.04 on mine. It’s well documented and easy to run NGINX on that to get a good webserver and robust re-streaming server all in one.

I did a ton of Google searching for information and came upon some reasonable tutorials. It was not very hard to get the re-streaming part of this going. Just forwarding streams of RTMP is not to hard and obviously doesn’t even take much computing power since the Ubuntu machine with 512mb of RAM can do it. It requires some simple config file modifications to the NGINX software with the rtmp module in it.

Also to get Facebook to work you have to add in a module called “stunnel”. That is pretty simple as well. The tutorial below should have most of what you need for Facebook, YouTube and any other service that requires only RTMP.

We’ve now streamed a few times to Facebook, YouTube and my HLS streaming server a few times. It works very well with this simple RTMP streaming server setup. It’s running well on my tiny $3.50 per month server just chugging along. Once the persistent stream keys are input in the server there isn’t any maintenance for it or any other reason to look at the configuration files. I’ll modify it sometime to add some of the statistics information that I have on my HLS server.

In addition to the below, you should fix the firewall to only allow through ports 1935, 80, maybe 440,990 and 1000. Depends on what else you do with it.

https://sites.google.com/view/facebook-rtmp-to-rtmps/home

If you’d like to see some of my nginx.conf file and other parts needed for this project see below. I’ll insert them here.

If you need to output an HLS or .m3u8 file, you need to go a little further. I made two servers so that I could get the majority working quickly and then I could modify the other without messing up the working one.

My second server needs to be made bigger to carry the load when we stream to the app. The app users are directly receiving the file from my server and they put a load on it. Also the creation of that HLS file is intensive alone, even if only one person is on the app. I’ll post about that second server in another post.

I hope this brings you as much of a feeling of accomplishment as it brought me. It also taught me a ton about linux and cloud servers.