Azuracast devours CPU, a big failure compared to the rest of the panels.

[alerickert]

Installation Method

One-Click Installation (Docker)

AzuraCast Release Channel

Rolling Release Channel

Current AzuraCast Version

#5fe95f4

What happened?

Hello everyone,

I'm an entrepreneur with 19 years of experience in the streaming industry. Recently, I've been testing AzuraCast with the intention of using it for production with large radio stations with over 10,000 real listeners.

Unfortunately, the results of my tests have been very poor in terms of CPU performance.

We're using servers with 32 cores and 128GB of RAM. As soon as we conduct stress tests, the CPU usage spikes above 90% and eventually crashes with a maximum of 20,000 listeners, causing interruptions. Even without AutoDJ, it can handle a bit more, but still encounters CPU issues. The CPU consumption is alarmingly high.

In comparison, with other panels like SonicPanel, 20,000 listeners consume about 3 CPUs and 20GB of RAM, whereas AzuraCast consumes 32 CPUs.

I would like to know if anyone is aware of this serious problem, as it unfortunately prevents us from using AzuraCast for large-scale operations.

Relevant log output

No response

[Vaalyn]

First of all: Coming into an open source projects repo (that is 99% maintained by one guy barely scraping by) and declaring it a "big failure" compared commercial projects who have dedicated paid engineers and an incentive to cater to big businesses due to paid support contracts is disingenuous at best and generally outright rude. I don't expect people to not complain about issues they encounter (otherwise those issues couldn't be discovered and fixed) but I do expect them to have a modicum of respect for other peoples work / live and not casually 💩 on it.

Apart from that, if you want to use this software for such high volume operations and there is an issue with the CPU usage it would be mighty helpful if we can get more than a "uses too much / all cpu", like actually information about:

• Which processes are using the cpu the most?
• How are you connecting the listeners?
  • Directly to the IceCast/ShoutCast ports?
  • Through the NGINX proxy?
  • Or through HLS?
• Which streaming formats are you using?
• Are the listeners only using the stream or the public page?

[alerickert]

I never declared it as a failure, what are you saying?

I find AzuraCast to be great, especially considering it's free. I would even pay for it if it worked properly in terms of resource usage. That's all.

I'm not questioning the talent and brilliance of AzuraCast, so don't say things I didn't say.

What I'm mentioning here is that AzuraCast has a very serious problem with resource consumption. The CPU usage spikes, and this is the negative aspect. However, from what I've tested so far, everything else seems to be working fine, although I haven't had a chance to thoroughly examine the statistics.

Listeners connect directly to the SSL URL at the mount point (url:port/mount) with Icecast, 48 AAC Plus, both with and without AutoDJ, and the results are the same. Initially, I thought it was Liquidsoap, but it's not. We also tested with 24kbps, with the same outcome.

HLS is disabled.

P.S.: SonicPanel is also not a software I like; in fact, it has poor statistics and design. So far, the one that works best for us is MediaCP.

And our idea was to migrate some large clients who use SonicPanel to AzuraCast.

[BusterNeece]

To be fair, the title of this issue is:

a big failure compared to the rest of the panels

So...we have every right to insist that users be more respectful and polite than that.

If listeners are connecting directly to the Icecast stream, then there are two possible explanations for excessive CPU consumption: it could be a result of the initial connection calling a PHP script to determine if the user is geographically banned in the panel (which would mean that the CPU consumption only spikes briefly during connection, and wouldn't be sustainably high, so as long as your listeners don't connect all at once like that, it won't be a problem) or it could be an optimization issue upstream with the Icecast-KH fork that we use for broadcasting.

It would be important in this case to determine whether the CPU spike is just upon initial connection of new listeners or whether it's sustained across their listening time, which it absolutely shouldn't be as Icecast is quite well optimized for that purpose.

[alerickert]

It might be a poor translation, as English is not my native language and I also use translators to assist me.

What we do is generate load through proxies, simulating real listeners, capable of making multiple connections at once, generating peaks of over 50,000 "real" listeners. However, as soon as we start, the CPU rapidly increases, and after reaching 15,000 to 20,000 listeners, CPU problems and interruptions begin.

What other tests could we perform? If you're willing and you have a VPS or dedicated server with Azura, we could conduct a test, and perhaps you could determine what causes AzuraCast to increase CPU usage.

[RM-FM]

@alerickert We also did some load tests in the past, albeit not with that many connections. However, such a test should build up the connections in stages. E.g. 50 per second until the target connection number is reached.

[alerickert]

These tests are the same as those we do for:
MediaCP
SonicPanel
CentovaCast
EverestCast

We did exactly the same for AzuraCast. In which we noticed this abnormality of resources, which we had never seen before. While we do notice that each panel supports more or less listeners, according to bandwidth and tuning specific to each panel's systems, we do not see 100% resource loads affecting all streams.

[BusterNeece]

@alerickert It's not useful to us to just tell us that other panels (especially closed-source, commercial ones) do something better than we do. I would avoid trying to put us up against the "competition" when asking volunteers to work toward your issue for free.

If you can dig deeper and find some information (like via htop) to see what is actually consuming the excessive CPU in this case, then we can investigate further. But please don't just tell us that other panels do it better.

[alerickert]

What consumes the most is: nginx: worker process, and then icecast.xml. Any ideas? I'm attaching a video of a test on a VPS server with 8 and 16 GB of RAM.

Video with an average traffic load: https://go.screenpal.com/watch/cZf1FMVs350

[Vaalyn]

@alerickert just to be sure, was this test directly connected to the IceCast port like this https://<domain>:<port> or to the listen URL that looks like this https://<domain>/listen/<station>/<mount>.mp3?

Also do you have any of these settings enabled on your installation?

[alerickert]

The options you mention are all disabled. Same test now at https://domain:port/mount.

[Vaalyn]

🤔 @BusterNeece do we have any other connection between IceCast and the PHP processes apart from the access callback for the above mentioned settings?

I've checked this on my test server and the respective authentication config in IceCast that would do the callbacks for those are not being put into the IceCast config file when they are disabled so if those are not enabled in this case here there has to be some other part that is getting flooded by the IceCast listeners.

[BusterNeece]

@Vaalyn I don't think there's any other explanation on our end, this might very well be something going on with Icecast-KH itself.

[Vaalyn]

@Vaalyn I don't think there's any other explanation on our end, this might very well be something going on with Icecast-KH itself.

I'd understand and agree on it being in Icecast-KH IF it was only the IceCast processes but since the PHP & NGINX processes are also being run quite high on that screenshot (on direct to IceCast port connections) there has to be some kind of link between Icecast and the web server that's being involved here and I'd guess that if there is indeed such a link then the Icecast CPU usage might partially come from waiting time for a response from the web server process 🤔

@alerickert could you maybe share the Icecast config from the testing server just to be sure I'm not missing anything while testing on my own server?

You can find it in the station under Logs -> Icecast Configuration

[alerickert]

hi, yes:

    <location>AzuraCast</location>
    <admin>icemaster@localhost</admin>
    <hostname>az.instream.audio</hostname>
    <limits>
        <clients>200000</clients>
        <sources>1</sources>
        <queue-size>524288</queue-size>
        <client-timeout>30</client-timeout>
        <header-timeout>15</header-timeout>
        <source-timeout>10</source-timeout>
        <burst-size>65535</burst-size>
    </limits>
    <authentication>
        <source-password>(PASSWORD)</source-password>
        <relay-password>(PASSWORD)</relay-password>
        <admin-user>admin</admin-user>
        <admin-password>(PASSWORD)</admin-password>
    </authentication>
    <listen-socket>
        <port>8000</port>
    </listen-socket>
    <mount type="normal">
        <mount-name>/radio</mount-name>
        <charset>UTF8</charset>
        <stream-name>Server Test</stream-name>
        <listenurl>https://az.instream.audio/listen/server_test/radio</listenurl>
        <fallback-mount>/fallback-[48].aac</fallback-mount>
        <fallback-override>1</fallback-override>
    </mount>
    <fileserve>1</fileserve>
    <paths>
        <basedir>/usr/local/share/icecast</basedir>
        <logdir>/var/azuracast/stations/server_test/config</logdir>
        <webroot>/usr/local/share/icecast/web</webroot>
        <adminroot>/usr/local/share/icecast/admin</adminroot>
        <pidfile>/var/azuracast/stations/server_test/config/icecast.pid</pidfile>
        <alias source="/" dest="/status.xsl"/>
        <ssl-private-key>/var/azuracast/storage/acme/ssl.key</ssl-private-key>
        <ssl-certificate>/var/azuracast/storage/acme/ssl.crt</ssl-certificate>
        <ssl-allowed-ciphers>ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS</ssl-allowed-ciphers>
        <deny-ip>/var/azuracast/stations/server_test/config/ip_bans.txt</deny-ip>
        <deny-agents>/var/azuracast/stations/server_test/config/user_agent_bans.txt</deny-agents>
        <x-forwarded-for>127.0.0.1</x-forwarded-for>
    </paths>
    <logging>
        <accesslog>icecast_access.log</accesslog>
        <errorlog>/dev/stderr</errorlog>
        <loglevel>2</loglevel>
        <logsize>10000</logsize>
    </logging>
    <security>
        <chroot>0</chroot>
    </security>
</icecast>

[Moonbase59]

I’ve not been testing with 200,000 listeners, but 5,000 max. per station, and the icecasts never took more than a few % CPU. It is known, though, that Icecast doesn’t like to connect/disconnect thousands of connections at exactly the same time. Testing scripts should take care of that. Mine uses a 0.1s delay between each connect, for example. Plus, of course you have to have enough bandwidth!

I’m wondering why all your three Icecasts refer to the exact same config file:

/var/azuracast/stations/server_test/config/icecast.xml

In my setup, one Icecast per station runs, and refers to a per-station icecast config.
Could that be the problem?

(About 300 listeners at the time of screenshot)